[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986536687152-17252-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 20:15:36 dist-test-slave-2hpg krb5kdc[17259](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 20:15:36 dist-test-slave-2hpg krb5kdc[17259](info): set up 2 sockets
Jun 20 20:15:36 dist-test-slave-2hpg krb5kdc[17259](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260620 20:15:37.826342 17252 mini_kdc.cc:121] Time spent starting KDC: real 1.048s user 0.000s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986536687152-17252-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986536687152-17252-0/krb5kdc/test-user.keytab.
Jun 20 20:15:37 dist-test-slave-2hpg krb5kdc[17259](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986537, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T20:15:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T20:15:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.217.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.217.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.217.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.62.keytab.
Entry for principal kudu/127.16.217.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.62.keytab.
I20260620 20:15:37.994745 17252 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
/tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986536687152-17252-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986536687152-17252-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986536687152-17252-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986536687152-17252-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.217.62:38091
--webserver_interface=127.16.217.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.62.keytab
--principal=kudu/127.16.217.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.217.20:45483
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.217.62:38091
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986536687152-17252-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986536687152-17252-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986536687152-17252-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:15:38.315141 17272 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:15:38.315505 17272 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:15:38.315676 17272 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:15:38.322804 17272 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:15:38.322962 17272 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:15:38.323071 17272 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:15:38.323175 17272 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:15:38.323244 17272 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:15:38.334010 17272 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.217.20:45483
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986536687152-17252-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986536687152-17252-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.217.62:38091
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.62.keytab
--principal=kudu/127.16.217.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.217.62:38091
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986536687152-17252-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.217.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986536687152-17252-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:15:38.335376 17272 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:15:38.336690 17272 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:15:38.348070 17277 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:15:38.348026 17278 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:15:38.348089 17280 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:15:38.349161 17272 server_base.cc:1034] running on GCE node
I20260620 20:15:38.349960 17272 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:15:38.351329 17272 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:15:38.352533 17272 hybrid_clock.cc:648] HybridClock initialized: now 1781986538352458 us; error 65 us; skew 500 ppm
I20260620 20:15:38.352993 17272 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:15:38 dist-test-slave-2hpg krb5kdc[17259](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986538, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.217.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:15:38.356112 17272 init.cc:377] Logged in from keytab as kudu/127.16.217.62@KRBTEST.COM (short username kudu)
I20260620 20:15:38.357573 17272 webserver.cc:460] Webserver started at http://127.16.217.62:36529/ using document root <none> and password file <none>
I20260620 20:15:38.358285 17272 fs_manager.cc:359] Metadata directory not provided
I20260620 20:15:38.358422 17272 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:15:38.358659 17272 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:15:38.362036 17272 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986536687152-17252-0/minicluster-data/master-0/data/instance:
uuid: "63de07000195467f8eb5b548d66f84b9"
format_stamp: "Formatted at 2026-06-20 20:15:38 on dist-test-slave-2hpg"
server_key: "4291d53bafc411d668928c8a1de299f1"
server_key_iv: "14702112b7f778392030a79d1c0f4e9a"
server_key_version: "encryptionkey@0"
I20260620 20:15:38.362730 17272 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986536687152-17252-0/minicluster-data/master-0/wal/instance:
uuid: "63de07000195467f8eb5b548d66f84b9"
format_stamp: "Formatted at 2026-06-20 20:15:38 on dist-test-slave-2hpg"
server_key: "4291d53bafc411d668928c8a1de299f1"
server_key_iv: "14702112b7f778392030a79d1c0f4e9a"
server_key_version: "encryptionkey@0"
I20260620 20:15:38.367812 17272 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 20:15:38.371245 17287 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:15:38.372615 17272 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 20:15:38.372797 17272 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986536687152-17252-0/minicluster-data/master-0/data,/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986536687152-17252-0/minicluster-data/master-0/wal
uuid: "63de07000195467f8eb5b548d66f84b9"
format_stamp: "Formatted at 2026-06-20 20:15:38 on dist-test-slave-2hpg"
server_key: "4291d53bafc411d668928c8a1de299f1"
server_key_iv: "14702112b7f778392030a79d1c0f4e9a"
server_key_version: "encryptionkey@0"
I20260620 20:15:38.373049 17272 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986536687152-17252-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986536687152-17252-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986536687152-17252-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:15:38.392755 17272 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:15:38.396752 17272 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:15:38.397034 17272 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:15:38.416002 17272 rpc_server.cc:307] RPC server started. Bound to: 127.16.217.62:38091
I20260620 20:15:38.416023 17338 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.217.62:38091 every 8 connection(s)
I20260620 20:15:38.417359 17272 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986536687152-17252-0/minicluster-data/master-0/data/info.pb
I20260620 20:15:38.420601 17252 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu as pid 17272
I20260620 20:15:38.420900 17252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986536687152-17252-0/minicluster-data/master-0/wal/instance
I20260620 20:15:38.421026 17339 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:15:38.422832 17252 external_mini_cluster.cc:1428] Setting key 68bbff1185ee3bfc42b8a6a037c8b3db
I20260620 20:15:38.432516 17339 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 63de07000195467f8eb5b548d66f84b9: Bootstrap starting.
I20260620 20:15:38.435621 17339 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 63de07000195467f8eb5b548d66f84b9: Neither blocks nor log segments found. Creating new log.
I20260620 20:15:38.437217 17339 log.cc:824] T 00000000000000000000000000000000 P 63de07000195467f8eb5b548d66f84b9: Log is configured to *not* fsync() on all Append() calls
I20260620 20:15:38.440074 17339 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 63de07000195467f8eb5b548d66f84b9: No bootstrap required, opened a new log
I20260620 20:15:38.448127 17339 raft_consensus.cc:348] T 00000000000000000000000000000000 P 63de07000195467f8eb5b548d66f84b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63de07000195467f8eb5b548d66f84b9" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 38091 } }
I20260620 20:15:38.448483 17339 raft_consensus.cc:374] T 00000000000000000000000000000000 P 63de07000195467f8eb5b548d66f84b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:15:38.448589 17339 raft_consensus.cc:729] T 00000000000000000000000000000000 P 63de07000195467f8eb5b548d66f84b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63de07000195467f8eb5b548d66f84b9, State: Initialized, Role: FOLLOWER
I20260620 20:15:38.449225 17339 consensus_queue.cc:260] T 00000000000000000000000000000000 P 63de07000195467f8eb5b548d66f84b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63de07000195467f8eb5b548d66f84b9" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 38091 } }
I20260620 20:15:38.449462 17339 raft_consensus.cc:388] T 00000000000000000000000000000000 P 63de07000195467f8eb5b548d66f84b9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 20:15:38.449602 17339 raft_consensus.cc:482] T 00000000000000000000000000000000 P 63de07000195467f8eb5b548d66f84b9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 20:15:38.449783 17339 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 63de07000195467f8eb5b548d66f84b9 [term 0 FOLLOWER]: Advancing to term 1
Jun 20 20:15:38 dist-test-slave-2hpg krb5kdc[17259](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986537, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.217.62@KRBTEST.COM
I20260620 20:15:38.452716 17339 raft_consensus.cc:504] T 00000000000000000000000000000000 P 63de07000195467f8eb5b548d66f84b9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63de07000195467f8eb5b548d66f84b9" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 38091 } }
I20260620 20:15:38.453265 17339 leader_election.cc:302] T 00000000000000000000000000000000 P 63de07000195467f8eb5b548d66f84b9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 63de07000195467f8eb5b548d66f84b9; no voters:
I20260620 20:15:38.454056 17339 leader_election.cc:288] T 00000000000000000000000000000000 P 63de07000195467f8eb5b548d66f84b9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 20:15:38.454202 17344 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 63de07000195467f8eb5b548d66f84b9 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:15:38.454838 17344 raft_consensus.cc:686] T 00000000000000000000000000000000 P 63de07000195467f8eb5b548d66f84b9 [term 1 LEADER]: Becoming Leader. State: Replica: 63de07000195467f8eb5b548d66f84b9, State: Running, Role: LEADER
I20260620 20:15:38.455410 17344 consensus_queue.cc:237] T 00000000000000000000000000000000 P 63de07000195467f8eb5b548d66f84b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63de07000195467f8eb5b548d66f84b9" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 38091 } }
I20260620 20:15:38.457151 17344 sys_catalog.cc:455] T 00000000000000000000000000000000 P 63de07000195467f8eb5b548d66f84b9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 63de07000195467f8eb5b548d66f84b9. Latest consensus state: current_term: 1 leader_uuid: "63de07000195467f8eb5b548d66f84b9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63de07000195467f8eb5b548d66f84b9" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 38091 } } }
I20260620 20:15:38.457422 17344 sys_catalog.cc:458] T 00000000000000000000000000000000 P 63de07000195467f8eb5b548d66f84b9 [sys.catalog]: This master's current role is: LEADER
I20260620 20:15:38.458047 17346 sys_catalog.cc:455] T 00000000000000000000000000000000 P 63de07000195467f8eb5b548d66f84b9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "63de07000195467f8eb5b548d66f84b9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63de07000195467f8eb5b548d66f84b9" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 38091 } } }
I20260620 20:15:38.458878 17346 sys_catalog.cc:458] T 00000000000000000000000000000000 P 63de07000195467f8eb5b548d66f84b9 [sys.catalog]: This master's current role is: LEADER
I20260620 20:15:38.464521 17341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:38.426466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37300 (local address 127.16.217.62:38091)
0620 20:15:38.427590 (+ 1124us) server_negotiation.cc:206] Beginning negotiation
0620 20:15:38.427600 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 20:15:38.428779 (+ 1179us) server_negotiation.cc:407] Connection header received
0620 20:15:38.437765 (+ 8986us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:15:38.437794 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:15:38.438280 (+ 486us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:15:38.438708 (+ 428us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:15:38.441441 (+ 2733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:38.442823 (+ 1382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:38.444417 (+ 1594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:38.445076 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:38.445271 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:15:38.451912 (+ 6641us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:15:38.451923 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:15:38.451935 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:15:38.451968 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:15:38.455944 (+ 3976us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:15:38.457185 (+ 1241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:15:38.457202 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:15:38.457213 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:15:38.457488 (+ 275us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:15:38.458275 (+ 787us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:15:38.458284 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:15:38.458290 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:15:38.459195 (+ 905us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:15:38.459588 (+ 393us) server_negotiation.cc:1035] Waiting for connection context
0620 20:15:38.462555 (+ 2967us) server_negotiation.cc:299] Negotiation successful
0620 20:15:38.463086 (+ 531us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":932,"thread_start_us":728,"threads_started":1}
I20260620 20:15:38.465785 17339 sys_catalog.cc:564] T 00000000000000000000000000000000 P 63de07000195467f8eb5b548d66f84b9 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:15:38.471750 17348 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 20:15:38.478849 17348 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 20:15:38.488158 17348 catalog_manager.cc:1347] Generated new cluster ID: 17d5a0670d094fa79f20f301193b85fd
I20260620 20:15:38.488281 17348 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 20:15:38.521943 17348 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 20:15:38.523883 17348 catalog_manager.cc:1504] Loading token signing keys...
I20260620 20:15:38.532971 17348 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 63de07000195467f8eb5b548d66f84b9: Generated new TSK 0
I20260620 20:15:38.534185 17348 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.217.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.217.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.217.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.1.keytab.
Entry for principal kudu/127.16.217.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.1.keytab.
I20260620 20:15:38.583566 17252 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
/tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986536687152-17252-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986536687152-17252-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986536687152-17252-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986536687152-17252-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.217.1:0
--local_ip_for_outbound_sockets=127.16.217.1
--webserver_interface=127.16.217.1
--webserver_port=0
--tserver_master_addrs=127.16.217.62:38091
--keytab_file=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.1.keytab
--principal=kudu/127.16.217.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.217.20:45483
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986536687152-17252-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986536687152-17252-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986536687152-17252-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:15:38.988116 17365 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:15:38.988420 17365 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:15:38.988598 17365 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:15:38.996387 17365 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:15:38.996492 17365 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:15:38.996606 17365 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.217.1
I20260620 20:15:39.006403 17365 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.217.20:45483
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986536687152-17252-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986536687152-17252-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.1.keytab
--principal=kudu/127.16.217.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.217.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986536687152-17252-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.217.1
--webserver_port=0
--tserver_master_addrs=127.16.217.62:38091
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.217.1
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986536687152-17252-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:15:39.007632 17365 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:15:39.008922 17365 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:15:39.022511 17373 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:15:39.022478 17371 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:15:39.022552 17370 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:15:39.027215 17365 server_base.cc:1034] running on GCE node
I20260620 20:15:39.027810 17365 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:15:39.028743 17365 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:15:39.029973 17365 hybrid_clock.cc:648] HybridClock initialized: now 1781986539029904 us; error 55 us; skew 500 ppm
I20260620 20:15:39.030303 17365 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:15:39 dist-test-slave-2hpg krb5kdc[17259](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986539, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.217.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:15:39.033463 17365 init.cc:377] Logged in from keytab as kudu/127.16.217.1@KRBTEST.COM (short username kudu)
I20260620 20:15:39.035025 17365 webserver.cc:460] Webserver started at http://127.16.217.1:34217/ using document root <none> and password file <none>
I20260620 20:15:39.035729 17365 fs_manager.cc:359] Metadata directory not provided
I20260620 20:15:39.035868 17365 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:15:39.036175 17365 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:15:39.039089 17365 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986536687152-17252-0/minicluster-data/ts-0/data/instance:
uuid: "0d8338cc335b4269b6f7ba0602d19081"
format_stamp: "Formatted at 2026-06-20 20:15:39 on dist-test-slave-2hpg"
server_key: "5047cd2bf6793c58f064214fa636db80"
server_key_iv: "93f46e3f9c41d7a3962aa211b4b3cb98"
server_key_version: "encryptionkey@0"
I20260620 20:15:39.039827 17365 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986536687152-17252-0/minicluster-data/ts-0/wal/instance:
uuid: "0d8338cc335b4269b6f7ba0602d19081"
format_stamp: "Formatted at 2026-06-20 20:15:39 on dist-test-slave-2hpg"
server_key: "5047cd2bf6793c58f064214fa636db80"
server_key_iv: "93f46e3f9c41d7a3962aa211b4b3cb98"
server_key_version: "encryptionkey@0"
I20260620 20:15:39.044723 17365 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260620 20:15:39.048094 17380 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:15:39.049494 17365 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260620 20:15:39.049685 17365 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986536687152-17252-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986536687152-17252-0/minicluster-data/ts-0/wal
uuid: "0d8338cc335b4269b6f7ba0602d19081"
format_stamp: "Formatted at 2026-06-20 20:15:39 on dist-test-slave-2hpg"
server_key: "5047cd2bf6793c58f064214fa636db80"
server_key_iv: "93f46e3f9c41d7a3962aa211b4b3cb98"
server_key_version: "encryptionkey@0"
I20260620 20:15:39.049942 17365 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986536687152-17252-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986536687152-17252-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986536687152-17252-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:15:39.070204 17365 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:15:39.074519 17365 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:15:39.074817 17365 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:15:39.075790 17365 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:15:39.077206 17365 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:15:39.077301 17365 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:15:39.077453 17365 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:15:39.077523 17365 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:15:39.111012 17365 rpc_server.cc:307] RPC server started. Bound to: 127.16.217.1:43575
I20260620 20:15:39.111097 17492 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.217.1:43575 every 8 connection(s)
I20260620 20:15:39.112334 17365 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986536687152-17252-0/minicluster-data/ts-0/data/info.pb
I20260620 20:15:39.112799 17252 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu as pid 17365
I20260620 20:15:39.113082 17252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986536687152-17252-0/minicluster-data/ts-0/wal/instance
I20260620 20:15:39.113929 17252 external_mini_cluster.cc:1428] Setting key 7a6de701dc531672da4e0b658c1cf1aa
Jun 20 20:15:39 dist-test-slave-2hpg krb5kdc[17259](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986539, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.217.1@KRBTEST.COM for kudu/127.16.217.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.16.217.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.217.2@KRBTEST.COM" created.
I20260620 20:15:39.134663 17495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:39.115203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.217.1:57845 (local address 127.16.217.62:38091)
0620 20:15:39.115718 (+ 515us) server_negotiation.cc:206] Beginning negotiation
0620 20:15:39.115724 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:15:39.116847 (+ 1123us) server_negotiation.cc:407] Connection header received
0620 20:15:39.118704 (+ 1857us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:15:39.118709 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:15:39.118803 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:15:39.118926 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:15:39.120502 (+ 1576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:39.121212 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:39.123889 (+ 2677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:39.124277 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:39.124379 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:15:39.128976 (+ 4597us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:15:39.128982 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:15:39.128985 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:15:39.129019 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:15:39.131688 (+ 2669us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:15:39.132412 (+ 724us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:15:39.132418 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:15:39.132420 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:15:39.132539 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:15:39.132961 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:15:39.132966 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:15:39.132969 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:15:39.133237 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:15:39.133388 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0620 20:15:39.134229 (+ 841us) server_negotiation.cc:299] Negotiation successful
0620 20:15:39.134408 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":399,"thread_start_us":264,"threads_started":1}
I20260620 20:15:39.136320 17496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:39.115868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:38091 (local address 127.16.217.1:57845)
0620 20:15:39.116700 (+ 832us) negotiation.cc:107] Waiting for socket to connect
0620 20:15:39.116744 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 20:15:39.118448 (+ 1704us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:15:39.119162 (+ 714us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:15:39.119171 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:15:39.119692 (+ 521us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:15:39.120301 (+ 609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:15:39.120330 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:15:39.121372 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:15:39.121380 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:15:39.123740 (+ 2360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:15:39.123757 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:15:39.124947 (+ 1190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:15:39.124956 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:15:39.125273 (+ 317us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:15:39.126028 (+ 755us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:15:39.126066 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:15:39.128743 (+ 2677us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:15:39.131828 (+ 3085us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:15:39.131838 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:15:39.131855 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:15:39.132279 (+ 424us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:15:39.132650 (+ 371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:15:39.132655 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:15:39.132659 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:15:39.132845 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:15:39.133372 (+ 527us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:15:39.133385 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:15:39.133728 (+ 343us) client_negotiation.cc:769] Sending connection context
0620 20:15:39.134109 (+ 381us) client_negotiation.cc:240] Negotiation successful
0620 20:15:39.134694 (+ 585us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":582,"thread_start_us":340,"threads_started":1}
I20260620 20:15:39.138491 17493 heartbeater.cc:347] Connected to a master server at 127.16.217.62:38091
I20260620 20:15:39.138864 17493 heartbeater.cc:464] Registering TS with master...
I20260620 20:15:39.139919 17493 heartbeater.cc:511] Master 127.16.217.62:38091 requested a full tablet report, sending...
I20260620 20:15:39.142233 17304 ts_manager.cc:194] Registered new tserver with Master: 0d8338cc335b4269b6f7ba0602d19081 (127.16.217.1:43575)
I20260620 20:15:39.143692 17304 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.217.1@KRBTEST.COM'} at 127.16.217.1:57845
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.217.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.2.keytab.
Entry for principal kudu/127.16.217.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.2.keytab.
I20260620 20:15:39.157204 17252 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
/tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986536687152-17252-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986536687152-17252-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986536687152-17252-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986536687152-17252-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.217.2:0
--local_ip_for_outbound_sockets=127.16.217.2
--webserver_interface=127.16.217.2
--webserver_port=0
--tserver_master_addrs=127.16.217.62:38091
--keytab_file=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.2.keytab
--principal=kudu/127.16.217.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.217.20:45483
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986536687152-17252-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986536687152-17252-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986536687152-17252-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:15:39.476883 17499 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:15:39.477211 17499 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:15:39.477397 17499 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:15:39.484586 17499 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:15:39.484683 17499 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:15:39.484862 17499 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.217.2
I20260620 20:15:39.494380 17499 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.217.20:45483
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986536687152-17252-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986536687152-17252-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.2.keytab
--principal=kudu/127.16.217.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.217.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986536687152-17252-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.217.2
--webserver_port=0
--tserver_master_addrs=127.16.217.62:38091
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.217.2
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986536687152-17252-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:15:39.496130 17499 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:15:39.497481 17499 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:15:39.511112 17507 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:15:39.511076 17505 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:15:39.511160 17504 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:15:39.517184 17499 server_base.cc:1034] running on GCE node
I20260620 20:15:39.517803 17499 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:15:39.518735 17499 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:15:39.519963 17499 hybrid_clock.cc:648] HybridClock initialized: now 1781986539519927 us; error 49 us; skew 500 ppm
I20260620 20:15:39.520268 17499 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:15:39 dist-test-slave-2hpg krb5kdc[17259](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986539, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.217.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:15:39.526516 17499 init.cc:377] Logged in from keytab as kudu/127.16.217.2@KRBTEST.COM (short username kudu)
I20260620 20:15:39.527900 17499 webserver.cc:460] Webserver started at http://127.16.217.2:35423/ using document root <none> and password file <none>
I20260620 20:15:39.528630 17499 fs_manager.cc:359] Metadata directory not provided
I20260620 20:15:39.528769 17499 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:15:39.529069 17499 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:15:39.531944 17499 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986536687152-17252-0/minicluster-data/ts-1/data/instance:
uuid: "f572890daf9142e9a2bf4d34aec39168"
format_stamp: "Formatted at 2026-06-20 20:15:39 on dist-test-slave-2hpg"
server_key: "d305b03050ceb2597d75d60c1e2511a7"
server_key_iv: "341c659c6a060a0e8bba3db246384a7b"
server_key_version: "encryptionkey@0"
I20260620 20:15:39.532673 17499 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986536687152-17252-0/minicluster-data/ts-1/wal/instance:
uuid: "f572890daf9142e9a2bf4d34aec39168"
format_stamp: "Formatted at 2026-06-20 20:15:39 on dist-test-slave-2hpg"
server_key: "d305b03050ceb2597d75d60c1e2511a7"
server_key_iv: "341c659c6a060a0e8bba3db246384a7b"
server_key_version: "encryptionkey@0"
I20260620 20:15:39.537523 17499 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.006s
I20260620 20:15:39.540941 17514 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:15:39.542285 17499 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 20:15:39.542454 17499 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986536687152-17252-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986536687152-17252-0/minicluster-data/ts-1/wal
uuid: "f572890daf9142e9a2bf4d34aec39168"
format_stamp: "Formatted at 2026-06-20 20:15:39 on dist-test-slave-2hpg"
server_key: "d305b03050ceb2597d75d60c1e2511a7"
server_key_iv: "341c659c6a060a0e8bba3db246384a7b"
server_key_version: "encryptionkey@0"
I20260620 20:15:39.542651 17499 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986536687152-17252-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986536687152-17252-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986536687152-17252-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:15:39.558023 17499 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:15:39.562301 17499 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:15:39.562580 17499 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:15:39.563510 17499 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:15:39.564749 17499 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:15:39.564851 17499 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:15:39.564996 17499 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:15:39.565073 17499 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:15:39.598718 17499 rpc_server.cc:307] RPC server started. Bound to: 127.16.217.2:44667
I20260620 20:15:39.598749 17626 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.217.2:44667 every 8 connection(s)
I20260620 20:15:39.600010 17499 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986536687152-17252-0/minicluster-data/ts-1/data/info.pb
I20260620 20:15:39.603057 17252 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu as pid 17499
I20260620 20:15:39.603286 17252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986536687152-17252-0/minicluster-data/ts-1/wal/instance
I20260620 20:15:39.603732 17252 external_mini_cluster.cc:1428] Setting key f92f9a1a7ae49873575ffc26340f3b8d
Jun 20 20:15:39 dist-test-slave-2hpg krb5kdc[17259](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986539, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.217.2@KRBTEST.COM for kudu/127.16.217.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.16.217.3@KRBTEST.COM; defaulting to no policy
I20260620 20:15:39.621634 17495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:39.603206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.217.2:39539 (local address 127.16.217.62:38091)
0620 20:15:39.603437 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0620 20:15:39.603446 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 20:15:39.604638 (+ 1192us) server_negotiation.cc:407] Connection header received
0620 20:15:39.606567 (+ 1929us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:15:39.606576 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:15:39.606679 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:15:39.606795 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:15:39.608203 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:39.609325 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:39.610950 (+ 1625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:39.611370 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:39.611464 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:15:39.615472 (+ 4008us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:15:39.615477 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:15:39.615480 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:15:39.615514 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:15:39.618561 (+ 3047us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:15:39.619263 (+ 702us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:15:39.619273 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:15:39.619275 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:15:39.619390 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:15:39.619862 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:15:39.619867 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:15:39.619870 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:15:39.620133 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:15:39.620296 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0620 20:15:39.621061 (+ 765us) server_negotiation.cc:299] Negotiation successful
0620 20:15:39.621279 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.217.3@KRBTEST.COM" created.
I20260620 20:15:39.622768 17629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:39.603667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:38091 (local address 127.16.217.2:39539)
0620 20:15:39.604474 (+ 807us) negotiation.cc:107] Waiting for socket to connect
0620 20:15:39.604523 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0620 20:15:39.606312 (+ 1789us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:15:39.607028 (+ 716us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:15:39.607036 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:15:39.607513 (+ 477us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:15:39.608049 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:15:39.608064 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:15:39.609471 (+ 1407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:15:39.609475 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:15:39.610778 (+ 1303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:15:39.610795 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:15:39.611579 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:15:39.611588 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:15:39.611913 (+ 325us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:15:39.612597 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:15:39.612629 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:15:39.615247 (+ 2618us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:15:39.618706 (+ 3459us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:15:39.618715 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:15:39.618731 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:15:39.619140 (+ 409us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:15:39.619504 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:15:39.619510 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:15:39.619513 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:15:39.619744 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:15:39.620254 (+ 510us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:15:39.620262 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:15:39.620674 (+ 412us) client_negotiation.cc:769] Sending connection context
0620 20:15:39.620970 (+ 296us) client_negotiation.cc:240] Negotiation successful
0620 20:15:39.621488 (+ 518us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":556,"thread_start_us":344,"threads_started":1}
I20260620 20:15:39.624254 17627 heartbeater.cc:347] Connected to a master server at 127.16.217.62:38091
I20260620 20:15:39.624534 17627 heartbeater.cc:464] Registering TS with master...
I20260620 20:15:39.625283 17627 heartbeater.cc:511] Master 127.16.217.62:38091 requested a full tablet report, sending...
I20260620 20:15:39.626856 17304 ts_manager.cc:194] Registered new tserver with Master: f572890daf9142e9a2bf4d34aec39168 (127.16.217.2:44667)
I20260620 20:15:39.627672 17304 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.217.2@KRBTEST.COM'} at 127.16.217.2:39539
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.217.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.3.keytab.
Entry for principal kudu/127.16.217.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.3.keytab.
I20260620 20:15:39.644616 17252 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
/tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986536687152-17252-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986536687152-17252-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986536687152-17252-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986536687152-17252-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.217.3:0
--local_ip_for_outbound_sockets=127.16.217.3
--webserver_interface=127.16.217.3
--webserver_port=0
--tserver_master_addrs=127.16.217.62:38091
--keytab_file=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.3.keytab
--principal=kudu/127.16.217.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.217.20:45483
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986536687152-17252-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986536687152-17252-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986536687152-17252-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:15:39.966815 17632 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:15:39.967082 17632 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:15:39.967245 17632 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:15:39.974431 17632 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:15:39.974529 17632 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:15:39.974735 17632 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.217.3
I20260620 20:15:39.984716 17632 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.217.20:45483
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986536687152-17252-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986536687152-17252-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.3.keytab
--principal=kudu/127.16.217.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.217.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986536687152-17252-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.217.3
--webserver_port=0
--tserver_master_addrs=127.16.217.62:38091
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.217.3
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986536687152-17252-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:15:39.985970 17632 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:15:39.987287 17632 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:15:40.000219 17637 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:15:40.000180 17638 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:15:40.000268 17640 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:15:40.001415 17632 server_base.cc:1034] running on GCE node
I20260620 20:15:40.002101 17632 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:15:40.003036 17632 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:15:40.004256 17632 hybrid_clock.cc:648] HybridClock initialized: now 1781986540004207 us; error 49 us; skew 500 ppm
I20260620 20:15:40.004577 17632 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:15:40 dist-test-slave-2hpg krb5kdc[17259](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986540, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.217.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:15:40.007670 17632 init.cc:377] Logged in from keytab as kudu/127.16.217.3@KRBTEST.COM (short username kudu)
I20260620 20:15:40.009193 17632 webserver.cc:460] Webserver started at http://127.16.217.3:36393/ using document root <none> and password file <none>
I20260620 20:15:40.009999 17632 fs_manager.cc:359] Metadata directory not provided
I20260620 20:15:40.010146 17632 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:15:40.010462 17632 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:15:40.013401 17632 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986536687152-17252-0/minicluster-data/ts-2/data/instance:
uuid: "6ee6641e69694aa195f4a51dcc812c48"
format_stamp: "Formatted at 2026-06-20 20:15:40 on dist-test-slave-2hpg"
server_key: "b26ac6b870cf144581ba899b57459739"
server_key_iv: "44b54415935c9d2aa631ba4bbefb3f39"
server_key_version: "encryptionkey@0"
I20260620 20:15:40.014281 17632 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986536687152-17252-0/minicluster-data/ts-2/wal/instance:
uuid: "6ee6641e69694aa195f4a51dcc812c48"
format_stamp: "Formatted at 2026-06-20 20:15:40 on dist-test-slave-2hpg"
server_key: "b26ac6b870cf144581ba899b57459739"
server_key_iv: "44b54415935c9d2aa631ba4bbefb3f39"
server_key_version: "encryptionkey@0"
I20260620 20:15:40.019601 17632 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 20:15:40.023137 17647 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:15:40.024889 17632 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 20:15:40.025239 17632 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986536687152-17252-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986536687152-17252-0/minicluster-data/ts-2/wal
uuid: "6ee6641e69694aa195f4a51dcc812c48"
format_stamp: "Formatted at 2026-06-20 20:15:40 on dist-test-slave-2hpg"
server_key: "b26ac6b870cf144581ba899b57459739"
server_key_iv: "44b54415935c9d2aa631ba4bbefb3f39"
server_key_version: "encryptionkey@0"
I20260620 20:15:40.025569 17632 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986536687152-17252-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986536687152-17252-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986536687152-17252-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:15:40.042071 17632 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:15:40.046039 17632 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:15:40.046310 17632 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:15:40.047293 17632 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:15:40.048525 17632 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:15:40.048635 17632 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:15:40.048794 17632 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:15:40.048871 17632 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:15:40.082376 17632 rpc_server.cc:307] RPC server started. Bound to: 127.16.217.3:44501
I20260620 20:15:40.082405 17759 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.217.3:44501 every 8 connection(s)
I20260620 20:15:40.083659 17632 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986536687152-17252-0/minicluster-data/ts-2/data/info.pb
I20260620 20:15:40.089708 17252 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu as pid 17632
I20260620 20:15:40.090015 17252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986536687152-17252-0/minicluster-data/ts-2/wal/instance
I20260620 20:15:40.090487 17252 external_mini_cluster.cc:1428] Setting key 9840ec925ae53e6fab90a3b17d6fbd13
Jun 20 20:15:40 dist-test-slave-2hpg krb5kdc[17259](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986540, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.217.3@KRBTEST.COM for kudu/127.16.217.62@KRBTEST.COM
I20260620 20:15:40.104496 17495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:40.086212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.217.3:58913 (local address 127.16.217.62:38091)
0620 20:15:40.086422 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0620 20:15:40.086428 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:15:40.088062 (+ 1634us) server_negotiation.cc:407] Connection header received
0620 20:15:40.089874 (+ 1812us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:15:40.089880 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:15:40.089990 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:15:40.090115 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:15:40.091632 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:40.092756 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:40.094897 (+ 2141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:40.095275 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:40.095387 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:15:40.099248 (+ 3861us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:15:40.099253 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:15:40.099256 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:15:40.099285 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:15:40.101777 (+ 2492us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:15:40.102462 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:15:40.102466 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:15:40.102468 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:15:40.102586 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:15:40.102987 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:15:40.102990 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:15:40.102992 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:15:40.103225 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:15:40.103328 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 20:15:40.104067 (+ 739us) server_negotiation.cc:299] Negotiation successful
0620 20:15:40.104220 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 20:15:40.105803 17762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:40.086806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:38091 (local address 127.16.217.3:58913)
0620 20:15:40.087496 (+ 690us) negotiation.cc:107] Waiting for socket to connect
0620 20:15:40.087542 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 20:15:40.089546 (+ 2004us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:15:40.090344 (+ 798us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:15:40.090353 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:15:40.090923 (+ 570us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:15:40.091471 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:15:40.091487 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:15:40.093007 (+ 1520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:15:40.093016 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:15:40.094754 (+ 1738us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:15:40.094765 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:15:40.095476 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:15:40.095481 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:15:40.095669 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:15:40.096406 (+ 737us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:15:40.096430 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:15:40.099065 (+ 2635us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:15:40.101961 (+ 2896us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:15:40.101968 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:15:40.101980 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:15:40.102360 (+ 380us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:15:40.102683 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:15:40.102686 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:15:40.102688 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:15:40.102889 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:15:40.103360 (+ 471us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:15:40.103367 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:15:40.103687 (+ 320us) client_negotiation.cc:769] Sending connection context
0620 20:15:40.103964 (+ 277us) client_negotiation.cc:240] Negotiation successful
0620 20:15:40.104338 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":273,"threads_started":1}
I20260620 20:15:40.107203 17760 heartbeater.cc:347] Connected to a master server at 127.16.217.62:38091
I20260620 20:15:40.107475 17760 heartbeater.cc:464] Registering TS with master...
I20260620 20:15:40.108199 17760 heartbeater.cc:511] Master 127.16.217.62:38091 requested a full tablet report, sending...
I20260620 20:15:40.109493 17303 ts_manager.cc:194] Registered new tserver with Master: 6ee6641e69694aa195f4a51dcc812c48 (127.16.217.3:44501)
I20260620 20:15:40.110378 17303 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.217.3@KRBTEST.COM'} at 127.16.217.3:58913
I20260620 20:15:40.119279 17252 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 20:15:40 dist-test-slave-2hpg krb5kdc[17259](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986540, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
I20260620 20:15:40.147742 17493 heartbeater.cc:503] Master 127.16.217.62:38091 was elected leader, sending a full tablet report...
Jun 20 20:15:40 dist-test-slave-2hpg krb5kdc[17259](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986540, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.16.217.1@KRBTEST.COM
I20260620 20:15:40.164122 17767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:40.147757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35570 (local address 127.16.217.1:43575)
0620 20:15:40.148662 (+ 905us) server_negotiation.cc:206] Beginning negotiation
0620 20:15:40.148675 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 20:15:40.148700 (+ 25us) server_negotiation.cc:407] Connection header received
0620 20:15:40.148752 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:15:40.148758 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:15:40.148976 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:15:40.149094 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:15:40.150612 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:40.152475 (+ 1863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:40.153809 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:40.154493 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:40.154684 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:15:40.158808 (+ 4124us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:15:40.158818 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:15:40.158832 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:15:40.158870 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:15:40.161568 (+ 2698us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:15:40.162266 (+ 698us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:15:40.162273 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:15:40.162276 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:15:40.162389 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:15:40.162706 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:15:40.162709 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:15:40.162711 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:15:40.163025 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:15:40.163242 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0620 20:15:40.163596 (+ 354us) server_negotiation.cc:299] Negotiation successful
0620 20:15:40.163790 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":743,"thread_start_us":595,"threads_started":1}
W20260620 20:15:40.165396 17408 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35570
Jun 20 20:15:40 dist-test-slave-2hpg krb5kdc[17259](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986540, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 20:15:40 dist-test-slave-2hpg krb5kdc[17259](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986540, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.217.1@KRBTEST.COM
I20260620 20:15:40.209180 17767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:40.194706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35586 (local address 127.16.217.1:43575)
0620 20:15:40.194967 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0620 20:15:40.194972 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:15:40.195142 (+ 170us) server_negotiation.cc:407] Connection header received
0620 20:15:40.195323 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:15:40.195327 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:15:40.195420 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:15:40.195502 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:15:40.196937 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:40.197780 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:40.199059 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:40.199479 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:40.199574 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:15:40.203631 (+ 4057us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:15:40.203636 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:15:40.203640 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:15:40.203671 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:15:40.206987 (+ 3316us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:15:40.207579 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:15:40.207583 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:15:40.207585 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:15:40.207701 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:15:40.208002 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:15:40.208005 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:15:40.208008 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:15:40.208243 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:15:40.208418 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0620 20:15:40.208662 (+ 244us) server_negotiation.cc:299] Negotiation successful
0620 20:15:40.208818 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260620 20:15:40.210834 17252 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu pid 17365
I20260620 20:15:40.223865 17767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:40.211822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35600 (local address 127.16.217.1:43575)
0620 20:15:40.211986 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0620 20:15:40.211990 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:15:40.212411 (+ 421us) server_negotiation.cc:407] Connection header received
0620 20:15:40.212529 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:15:40.212533 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:15:40.212588 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:15:40.212673 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:15:40.213822 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:40.214632 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:40.215936 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:40.216345 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:40.216443 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:15:40.218746 (+ 2303us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:15:40.218751 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:15:40.218754 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:15:40.218781 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:15:40.221257 (+ 2476us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:15:40.221862 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:15:40.221908 (+ 46us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:15:40.221912 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:15:40.222065 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:15:40.222393 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:15:40.222396 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:15:40.222398 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:15:40.223032 (+ 634us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:15:40.223199 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0620 20:15:40.223462 (+ 263us) server_negotiation.cc:299] Negotiation successful
0620 20:15:40.223606 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 20:15:40.224681 17488 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:15:40.311250 17252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu with pid 17365
I20260620 20:15:40.326968 17252 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu pid 17499
Jun 20 20:15:40 dist-test-slave-2hpg krb5kdc[17259](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986540, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.217.2@KRBTEST.COM
I20260620 20:15:40.342522 17774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:40.327841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57522 (local address 127.16.217.2:44667)
0620 20:15:40.328345 (+ 504us) server_negotiation.cc:206] Beginning negotiation
0620 20:15:40.328351 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:15:40.328385 (+ 34us) server_negotiation.cc:407] Connection header received
0620 20:15:40.328443 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:15:40.328470 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:15:40.328681 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:15:40.328852 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:15:40.330093 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:40.331078 (+ 985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:40.332292 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:40.332848 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:40.332983 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:15:40.336925 (+ 3942us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:15:40.336935 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:15:40.336948 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:15:40.336978 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:15:40.339645 (+ 2667us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:15:40.340309 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:15:40.340321 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:15:40.340324 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:15:40.340448 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:15:40.340842 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:15:40.340848 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:15:40.340851 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:15:40.341185 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:15:40.341503 (+ 318us) server_negotiation.cc:1035] Waiting for connection context
0620 20:15:40.341792 (+ 289us) server_negotiation.cc:299] Negotiation successful
0620 20:15:40.342176 (+ 384us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":372,"thread_start_us":247,"threads_started":1}
I20260620 20:15:40.343813 17622 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:15:40.425496 17252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu with pid 17499
I20260620 20:15:40.447258 17252 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu pid 17632
Jun 20 20:15:40 dist-test-slave-2hpg krb5kdc[17259](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986540, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.217.3@KRBTEST.COM
I20260620 20:15:40.462697 17776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:40.448123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53718 (local address 127.16.217.3:44501)
0620 20:15:40.448686 (+ 563us) server_negotiation.cc:206] Beginning negotiation
0620 20:15:40.448692 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:15:40.448720 (+ 28us) server_negotiation.cc:407] Connection header received
0620 20:15:40.448778 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:15:40.448784 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:15:40.449001 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:15:40.449167 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:15:40.450454 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:40.451461 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:40.452713 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:40.453258 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:40.453394 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:15:40.457345 (+ 3951us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:15:40.457356 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:15:40.457370 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:15:40.457402 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:15:40.460159 (+ 2757us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:15:40.460781 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:15:40.460788 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:15:40.460790 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:15:40.460897 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:15:40.461241 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:15:40.461245 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:15:40.461247 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:15:40.461589 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:15:40.461816 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0620 20:15:40.462242 (+ 426us) server_negotiation.cc:299] Negotiation successful
0620 20:15:40.462432 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":425,"thread_start_us":272,"threads_started":1}
I20260620 20:15:40.463912 17755 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:15:40.544826 17252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu with pid 17632
I20260620 20:15:40.560988 17252 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu pid 17272
I20260620 20:15:40.562189 17334 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:15:40.662160 17252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu with pid 17272
2026-06-20T20:15:40Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (3936 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986536687152-17252-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 20:15:40 dist-test-slave-2hpg krb5kdc[17781](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 20:15:40 dist-test-slave-2hpg krb5kdc[17781](info): set up 2 sockets
Jun 20 20:15:40 dist-test-slave-2hpg krb5kdc[17781](info): commencing operation
krb5kdc: starting...
W20260620 20:15:41.746209 17252 mini_kdc.cc:121] Time spent starting KDC: real 1.042s user 0.000s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986536687152-17252-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986536687152-17252-0/krb5kdc/test-user.keytab.
Jun 20 20:15:41 dist-test-slave-2hpg krb5kdc[17781](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986541, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T20:15:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T20:15:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.217.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.217.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.217.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.62.keytab.
Entry for principal kudu/127.16.217.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.62.keytab.
I20260620 20:15:41.921483 17252 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
/tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986536687152-17252-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986536687152-17252-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986536687152-17252-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986536687152-17252-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.217.62:42591
--webserver_interface=127.16.217.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.62.keytab
--principal=kudu/127.16.217.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.217.20:45485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.217.62:42591
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986536687152-17252-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986536687152-17252-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986536687152-17252-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:15:42.236407 17794 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:15:42.236727 17794 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:15:42.236897 17794 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:15:42.243896 17794 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:15:42.244037 17794 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:15:42.244148 17794 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:15:42.244249 17794 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:15:42.244335 17794 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:15:42.253836 17794 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.217.20:45485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986536687152-17252-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986536687152-17252-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.217.62:42591
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.62.keytab
--principal=kudu/127.16.217.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.217.62:42591
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986536687152-17252-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.217.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986536687152-17252-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:15:42.255142 17794 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:15:42.256409 17794 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:15:42.268254 17799 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:15:42.268275 17800 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:15:42.268332 17802 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:15:42.268993 17794 server_base.cc:1034] running on GCE node
I20260620 20:15:42.269831 17794 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:15:42.271279 17794 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:15:42.272485 17794 hybrid_clock.cc:648] HybridClock initialized: now 1781986542272440 us; error 42 us; skew 500 ppm
I20260620 20:15:42.272850 17794 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:15:42 dist-test-slave-2hpg krb5kdc[17781](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986542, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.217.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:15:42.275991 17794 init.cc:377] Logged in from keytab as kudu/127.16.217.62@KRBTEST.COM (short username kudu)
I20260620 20:15:42.277498 17794 webserver.cc:460] Webserver started at http://127.16.217.62:39017/ using document root <none> and password file <none>
I20260620 20:15:42.278285 17794 fs_manager.cc:359] Metadata directory not provided
I20260620 20:15:42.278438 17794 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:15:42.278842 17794 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:15:42.282291 17794 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986536687152-17252-0/minicluster-data/master-0/data/instance:
uuid: "7fbb9f5f475d44e4a0c46de8f17f9e43"
format_stamp: "Formatted at 2026-06-20 20:15:42 on dist-test-slave-2hpg"
server_key: "75fa684281e99c8856c7bd46dcefbd91"
server_key_iv: "6ac9163430bb352ee95aa520095999f2"
server_key_version: "encryptionkey@0"
I20260620 20:15:42.283080 17794 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986536687152-17252-0/minicluster-data/master-0/wal/instance:
uuid: "7fbb9f5f475d44e4a0c46de8f17f9e43"
format_stamp: "Formatted at 2026-06-20 20:15:42 on dist-test-slave-2hpg"
server_key: "75fa684281e99c8856c7bd46dcefbd91"
server_key_iv: "6ac9163430bb352ee95aa520095999f2"
server_key_version: "encryptionkey@0"
I20260620 20:15:42.288321 17794 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260620 20:15:42.291754 17809 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:15:42.293077 17794 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 20:15:42.293254 17794 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986536687152-17252-0/minicluster-data/master-0/data,/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986536687152-17252-0/minicluster-data/master-0/wal
uuid: "7fbb9f5f475d44e4a0c46de8f17f9e43"
format_stamp: "Formatted at 2026-06-20 20:15:42 on dist-test-slave-2hpg"
server_key: "75fa684281e99c8856c7bd46dcefbd91"
server_key_iv: "6ac9163430bb352ee95aa520095999f2"
server_key_version: "encryptionkey@0"
I20260620 20:15:42.293463 17794 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986536687152-17252-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986536687152-17252-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986536687152-17252-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:15:42.312777 17794 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:15:42.317034 17794 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:15:42.317332 17794 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:15:42.336540 17794 rpc_server.cc:307] RPC server started. Bound to: 127.16.217.62:42591
I20260620 20:15:42.336541 17860 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.217.62:42591 every 8 connection(s)
I20260620 20:15:42.337986 17794 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986536687152-17252-0/minicluster-data/master-0/data/info.pb
I20260620 20:15:42.342038 17861 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:15:42.347009 17252 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu as pid 17794
I20260620 20:15:42.347239 17252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986536687152-17252-0/minicluster-data/master-0/wal/instance
I20260620 20:15:42.347870 17252 external_mini_cluster.cc:1428] Setting key 5fd04268abc3b6a27ced976cf6c597bb
I20260620 20:15:42.355945 17861 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7fbb9f5f475d44e4a0c46de8f17f9e43: Bootstrap starting.
I20260620 20:15:42.360222 17861 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7fbb9f5f475d44e4a0c46de8f17f9e43: Neither blocks nor log segments found. Creating new log.
I20260620 20:15:42.361428 17861 log.cc:824] T 00000000000000000000000000000000 P 7fbb9f5f475d44e4a0c46de8f17f9e43: Log is configured to *not* fsync() on all Append() calls
I20260620 20:15:42.364537 17861 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7fbb9f5f475d44e4a0c46de8f17f9e43: No bootstrap required, opened a new log
Jun 20 20:15:42 dist-test-slave-2hpg krb5kdc[17781](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986541, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.217.62@KRBTEST.COM
I20260620 20:15:42.372576 17861 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7fbb9f5f475d44e4a0c46de8f17f9e43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fbb9f5f475d44e4a0c46de8f17f9e43" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 42591 } }
I20260620 20:15:42.372958 17861 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7fbb9f5f475d44e4a0c46de8f17f9e43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:15:42.372987 17864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:42.349813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39048 (local address 127.16.217.62:42591)
0620 20:15:42.350856 (+ 1043us) server_negotiation.cc:206] Beginning negotiation
0620 20:15:42.350875 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0620 20:15:42.350941 (+ 66us) server_negotiation.cc:407] Connection header received
0620 20:15:42.353944 (+ 3003us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:15:42.353983 (+ 39us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:15:42.354528 (+ 545us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:15:42.355055 (+ 527us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:15:42.356794 (+ 1739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:42.359131 (+ 2337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:42.360420 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:42.361118 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:42.361319 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:15:42.365667 (+ 4348us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:15:42.365675 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:15:42.365689 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:15:42.365726 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:15:42.368827 (+ 3101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:15:42.369542 (+ 715us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:15:42.369561 (+ 19us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:15:42.369568 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:15:42.369734 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:15:42.370151 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:15:42.370156 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:15:42.370169 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:15:42.370684 (+ 515us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:15:42.371027 (+ 343us) server_negotiation.cc:1035] Waiting for connection context
0620 20:15:42.371533 (+ 506us) server_negotiation.cc:299] Negotiation successful
0620 20:15:42.371951 (+ 418us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":744,"thread_start_us":413,"threads_started":1}
I20260620 20:15:42.373085 17861 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7fbb9f5f475d44e4a0c46de8f17f9e43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fbb9f5f475d44e4a0c46de8f17f9e43, State: Initialized, Role: FOLLOWER
I20260620 20:15:42.373700 17861 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7fbb9f5f475d44e4a0c46de8f17f9e43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fbb9f5f475d44e4a0c46de8f17f9e43" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 42591 } }
I20260620 20:15:42.373960 17861 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7fbb9f5f475d44e4a0c46de8f17f9e43 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 20:15:42.374112 17861 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7fbb9f5f475d44e4a0c46de8f17f9e43 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 20:15:42.374284 17861 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7fbb9f5f475d44e4a0c46de8f17f9e43 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:15:42.376664 17861 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7fbb9f5f475d44e4a0c46de8f17f9e43 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fbb9f5f475d44e4a0c46de8f17f9e43" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 42591 } }
I20260620 20:15:42.377162 17861 leader_election.cc:302] T 00000000000000000000000000000000 P 7fbb9f5f475d44e4a0c46de8f17f9e43 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7fbb9f5f475d44e4a0c46de8f17f9e43; no voters:
I20260620 20:15:42.377682 17861 leader_election.cc:288] T 00000000000000000000000000000000 P 7fbb9f5f475d44e4a0c46de8f17f9e43 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 20:15:42.377815 17866 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7fbb9f5f475d44e4a0c46de8f17f9e43 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:15:42.379170 17866 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7fbb9f5f475d44e4a0c46de8f17f9e43 [term 1 LEADER]: Becoming Leader. State: Replica: 7fbb9f5f475d44e4a0c46de8f17f9e43, State: Running, Role: LEADER
I20260620 20:15:42.379778 17861 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7fbb9f5f475d44e4a0c46de8f17f9e43 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:15:42.379660 17866 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7fbb9f5f475d44e4a0c46de8f17f9e43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fbb9f5f475d44e4a0c46de8f17f9e43" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 42591 } }
I20260620 20:15:42.382728 17867 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7fbb9f5f475d44e4a0c46de8f17f9e43 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7fbb9f5f475d44e4a0c46de8f17f9e43" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fbb9f5f475d44e4a0c46de8f17f9e43" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 42591 } } }
I20260620 20:15:42.382800 17868 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7fbb9f5f475d44e4a0c46de8f17f9e43 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7fbb9f5f475d44e4a0c46de8f17f9e43. Latest consensus state: current_term: 1 leader_uuid: "7fbb9f5f475d44e4a0c46de8f17f9e43" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fbb9f5f475d44e4a0c46de8f17f9e43" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 42591 } } }
I20260620 20:15:42.383095 17867 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7fbb9f5f475d44e4a0c46de8f17f9e43 [sys.catalog]: This master's current role is: LEADER
I20260620 20:15:42.383179 17868 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7fbb9f5f475d44e4a0c46de8f17f9e43 [sys.catalog]: This master's current role is: LEADER
I20260620 20:15:42.386802 17874 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 20:15:42.391837 17874 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 20:15:42.402069 17874 catalog_manager.cc:1347] Generated new cluster ID: eb0c3547ef88409180f2827464a8971a
I20260620 20:15:42.402179 17874 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 20:15:42.413166 17874 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 20:15:42.414579 17874 catalog_manager.cc:1504] Loading token signing keys...
I20260620 20:15:42.426056 17874 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7fbb9f5f475d44e4a0c46de8f17f9e43: Generated new TSK 0
I20260620 20:15:42.427042 17874 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.217.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.217.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.217.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.1.keytab.
Entry for principal kudu/127.16.217.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.1.keytab.
I20260620 20:15:42.471495 17252 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
/tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986536687152-17252-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986536687152-17252-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986536687152-17252-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986536687152-17252-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.217.1:0
--local_ip_for_outbound_sockets=127.16.217.1
--webserver_interface=127.16.217.1
--webserver_port=0
--tserver_master_addrs=127.16.217.62:42591
--keytab_file=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.1.keytab
--principal=kudu/127.16.217.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.217.20:45485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986536687152-17252-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986536687152-17252-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986536687152-17252-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:15:42.817204 17887 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:15:42.817487 17887 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:15:42.817660 17887 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:15:42.824878 17887 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:15:42.824980 17887 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:15:42.825167 17887 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.217.1
I20260620 20:15:42.834699 17887 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.217.20:45485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986536687152-17252-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986536687152-17252-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.1.keytab
--principal=kudu/127.16.217.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.217.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986536687152-17252-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.217.1
--webserver_port=0
--tserver_master_addrs=127.16.217.62:42591
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.217.1
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986536687152-17252-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:15:42.835953 17887 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:15:42.837257 17887 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:15:42.850358 17895 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:15:42.850430 17893 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:15:42.850570 17892 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:15:42.851570 17887 server_base.cc:1034] running on GCE node
I20260620 20:15:42.852205 17887 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:15:42.853106 17887 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:15:42.854332 17887 hybrid_clock.cc:648] HybridClock initialized: now 1781986542854293 us; error 50 us; skew 500 ppm
I20260620 20:15:42.854674 17887 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:15:42 dist-test-slave-2hpg krb5kdc[17781](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986542, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.217.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:15:42.857715 17887 init.cc:377] Logged in from keytab as kudu/127.16.217.1@KRBTEST.COM (short username kudu)
I20260620 20:15:42.859411 17887 webserver.cc:460] Webserver started at http://127.16.217.1:38925/ using document root <none> and password file <none>
I20260620 20:15:42.860088 17887 fs_manager.cc:359] Metadata directory not provided
I20260620 20:15:42.860230 17887 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:15:42.860532 17887 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:15:42.863415 17887 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986536687152-17252-0/minicluster-data/ts-0/data/instance:
uuid: "b389338496ea4674878ed902795b3da7"
format_stamp: "Formatted at 2026-06-20 20:15:42 on dist-test-slave-2hpg"
server_key: "d3f9a0cdfe6b32f746388a82bd5e66e3"
server_key_iv: "9a3c74765978041f339d994d6a8a387c"
server_key_version: "encryptionkey@0"
I20260620 20:15:42.864179 17887 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986536687152-17252-0/minicluster-data/ts-0/wal/instance:
uuid: "b389338496ea4674878ed902795b3da7"
format_stamp: "Formatted at 2026-06-20 20:15:42 on dist-test-slave-2hpg"
server_key: "d3f9a0cdfe6b32f746388a82bd5e66e3"
server_key_iv: "9a3c74765978041f339d994d6a8a387c"
server_key_version: "encryptionkey@0"
I20260620 20:15:42.869221 17887 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 20:15:42.872725 17902 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:15:42.874164 17887 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 20:15:42.874289 17887 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986536687152-17252-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986536687152-17252-0/minicluster-data/ts-0/wal
uuid: "b389338496ea4674878ed902795b3da7"
format_stamp: "Formatted at 2026-06-20 20:15:42 on dist-test-slave-2hpg"
server_key: "d3f9a0cdfe6b32f746388a82bd5e66e3"
server_key_iv: "9a3c74765978041f339d994d6a8a387c"
server_key_version: "encryptionkey@0"
I20260620 20:15:42.874451 17887 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986536687152-17252-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986536687152-17252-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986536687152-17252-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:15:42.890942 17887 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:15:42.894783 17887 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:15:42.895018 17887 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:15:42.895939 17887 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:15:42.897190 17887 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:15:42.897264 17887 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:15:42.897418 17887 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:15:42.897464 17887 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:15:42.930475 17887 rpc_server.cc:307] RPC server started. Bound to: 127.16.217.1:32937
I20260620 20:15:42.930560 18014 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.217.1:32937 every 8 connection(s)
I20260620 20:15:42.931707 17887 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986536687152-17252-0/minicluster-data/ts-0/data/info.pb
I20260620 20:15:42.939921 17252 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu as pid 17887
I20260620 20:15:42.940143 17252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986536687152-17252-0/minicluster-data/ts-0/wal/instance
I20260620 20:15:42.940728 17252 external_mini_cluster.cc:1428] Setting key f9d38ae7d44118dd6c12a0a897744cc9
Jun 20 20:15:42 dist-test-slave-2hpg krb5kdc[17781](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986542, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.217.1@KRBTEST.COM for kudu/127.16.217.62@KRBTEST.COM
I20260620 20:15:42.951821 18017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:42.934121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.217.1:52681 (local address 127.16.217.62:42591)
0620 20:15:42.934549 (+ 428us) server_negotiation.cc:206] Beginning negotiation
0620 20:15:42.934553 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:15:42.935571 (+ 1018us) server_negotiation.cc:407] Connection header received
0620 20:15:42.937369 (+ 1798us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:15:42.937375 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:15:42.937474 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:15:42.937594 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:15:42.939111 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:42.939855 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:42.941580 (+ 1725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:42.942145 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:42.942279 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:15:42.945840 (+ 3561us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:15:42.945848 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:15:42.945856 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:15:42.945926 (+ 70us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:15:42.948651 (+ 2725us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:15:42.949559 (+ 908us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:15:42.949563 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:15:42.949565 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:15:42.949670 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:15:42.950163 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:15:42.950166 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:15:42.950168 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:15:42.950421 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:15:42.950528 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0620 20:15:42.951411 (+ 883us) server_negotiation.cc:299] Negotiation successful
0620 20:15:42.951565 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":325,"thread_start_us":217,"threads_started":1}
I20260620 20:15:42.953322 18018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:42.934741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:42591 (local address 127.16.217.1:52681)
0620 20:15:42.935410 (+ 669us) negotiation.cc:107] Waiting for socket to connect
0620 20:15:42.935446 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 20:15:42.937105 (+ 1659us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:15:42.937824 (+ 719us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:15:42.937832 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:15:42.938387 (+ 555us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:15:42.938941 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:15:42.938958 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:15:42.940061 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:15:42.940069 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:15:42.941401 (+ 1332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:15:42.941435 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:15:42.942344 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:15:42.942349 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:15:42.942648 (+ 299us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:15:42.943306 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:15:42.943327 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:15:42.945622 (+ 2295us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:15:42.948787 (+ 3165us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:15:42.948798 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:15:42.948818 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:15:42.949444 (+ 626us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:15:42.949774 (+ 330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:15:42.949781 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:15:42.949788 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:15:42.950056 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:15:42.950547 (+ 491us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:15:42.950556 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:15:42.951003 (+ 447us) client_negotiation.cc:769] Sending connection context
0620 20:15:42.951305 (+ 302us) client_negotiation.cc:240] Negotiation successful
0620 20:15:42.951859 (+ 554us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":250,"threads_started":1}
I20260620 20:15:42.954998 18015 heartbeater.cc:347] Connected to a master server at 127.16.217.62:42591
I20260620 20:15:42.955293 18015 heartbeater.cc:464] Registering TS with master...
I20260620 20:15:42.955986 18015 heartbeater.cc:511] Master 127.16.217.62:42591 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.16.217.2@KRBTEST.COM; defaulting to no policy
I20260620 20:15:42.958097 17826 ts_manager.cc:194] Registered new tserver with Master: b389338496ea4674878ed902795b3da7 (127.16.217.1:32937)
I20260620 20:15:42.959615 17826 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.217.1@KRBTEST.COM'} at 127.16.217.1:52681
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.217.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.217.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.2.keytab.
Entry for principal kudu/127.16.217.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.2.keytab.
I20260620 20:15:42.982337 17252 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
/tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986536687152-17252-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986536687152-17252-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986536687152-17252-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986536687152-17252-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.217.2:0
--local_ip_for_outbound_sockets=127.16.217.2
--webserver_interface=127.16.217.2
--webserver_port=0
--tserver_master_addrs=127.16.217.62:42591
--keytab_file=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.2.keytab
--principal=kudu/127.16.217.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.217.20:45485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986536687152-17252-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986536687152-17252-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986536687152-17252-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:15:43.294540 18021 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:15:43.294878 18021 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:15:43.295086 18021 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:15:43.302343 18021 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:15:43.302508 18021 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:15:43.302706 18021 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.217.2
I20260620 20:15:43.312511 18021 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.217.20:45485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986536687152-17252-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986536687152-17252-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.2.keytab
--principal=kudu/127.16.217.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.217.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986536687152-17252-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.217.2
--webserver_port=0
--tserver_master_addrs=127.16.217.62:42591
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.217.2
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986536687152-17252-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:15:43.313838 18021 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:15:43.315338 18021 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:15:43.328197 18026 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:15:43.328240 18027 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:15:43.328480 18029 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:15:43.329218 18021 server_base.cc:1034] running on GCE node
I20260620 20:15:43.329787 18021 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:15:43.330716 18021 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:15:43.331964 18021 hybrid_clock.cc:648] HybridClock initialized: now 1781986543331925 us; error 52 us; skew 500 ppm
I20260620 20:15:43.332264 18021 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:15:43 dist-test-slave-2hpg krb5kdc[17781](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986543, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.217.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:15:43.335294 18021 init.cc:377] Logged in from keytab as kudu/127.16.217.2@KRBTEST.COM (short username kudu)
I20260620 20:15:43.336715 18021 webserver.cc:460] Webserver started at http://127.16.217.2:41719/ using document root <none> and password file <none>
I20260620 20:15:43.337379 18021 fs_manager.cc:359] Metadata directory not provided
I20260620 20:15:43.337500 18021 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:15:43.337806 18021 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:15:43.340549 18021 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986536687152-17252-0/minicluster-data/ts-1/data/instance:
uuid: "85a41b286f0843c4a57a81fc7bcecf85"
format_stamp: "Formatted at 2026-06-20 20:15:43 on dist-test-slave-2hpg"
server_key: "bb2de89bfae6f45f134f10d33548cc39"
server_key_iv: "97c32cd4d6f5fbd8517536ba8fc4f605"
server_key_version: "encryptionkey@0"
I20260620 20:15:43.341305 18021 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986536687152-17252-0/minicluster-data/ts-1/wal/instance:
uuid: "85a41b286f0843c4a57a81fc7bcecf85"
format_stamp: "Formatted at 2026-06-20 20:15:43 on dist-test-slave-2hpg"
server_key: "bb2de89bfae6f45f134f10d33548cc39"
server_key_iv: "97c32cd4d6f5fbd8517536ba8fc4f605"
server_key_version: "encryptionkey@0"
I20260620 20:15:43.346253 18021 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260620 20:15:43.349684 18036 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:15:43.350917 18021 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 20:15:43.351068 18021 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986536687152-17252-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986536687152-17252-0/minicluster-data/ts-1/wal
uuid: "85a41b286f0843c4a57a81fc7bcecf85"
format_stamp: "Formatted at 2026-06-20 20:15:43 on dist-test-slave-2hpg"
server_key: "bb2de89bfae6f45f134f10d33548cc39"
server_key_iv: "97c32cd4d6f5fbd8517536ba8fc4f605"
server_key_version: "encryptionkey@0"
I20260620 20:15:43.351248 18021 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986536687152-17252-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986536687152-17252-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986536687152-17252-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:15:43.367151 18021 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:15:43.370754 18021 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:15:43.371042 18021 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:15:43.371953 18021 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:15:43.373207 18021 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:15:43.373307 18021 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:15:43.373462 18021 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:15:43.373535 18021 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:15:43.407045 18021 rpc_server.cc:307] RPC server started. Bound to: 127.16.217.2:39107
I20260620 20:15:43.407131 18148 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.217.2:39107 every 8 connection(s)
I20260620 20:15:43.408354 18021 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986536687152-17252-0/minicluster-data/ts-1/data/info.pb
I20260620 20:15:43.418352 17252 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu as pid 18021
I20260620 20:15:43.418563 17252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986536687152-17252-0/minicluster-data/ts-1/wal/instance
I20260620 20:15:43.419025 17252 external_mini_cluster.cc:1428] Setting key 9107c2b1d0ccde7539653af91f62e613
Jun 20 20:15:43 dist-test-slave-2hpg krb5kdc[17781](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986543, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.217.2@KRBTEST.COM for kudu/127.16.217.62@KRBTEST.COM
I20260620 20:15:43.428045 18017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:43.411044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.217.2:56071 (local address 127.16.217.62:42591)
0620 20:15:43.411252 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0620 20:15:43.411257 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:15:43.412377 (+ 1120us) server_negotiation.cc:407] Connection header received
0620 20:15:43.414188 (+ 1811us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:15:43.414194 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:15:43.414287 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:15:43.414406 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:15:43.415827 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:43.416541 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:43.418079 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:43.418500 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:43.418639 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:15:43.422581 (+ 3942us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:15:43.422586 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:15:43.422589 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:15:43.422620 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:15:43.425159 (+ 2539us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:15:43.425845 (+ 686us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:15:43.425851 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:15:43.425855 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:15:43.426024 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:15:43.426435 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:15:43.426440 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:15:43.426444 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:15:43.426695 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:15:43.426843 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0620 20:15:43.427488 (+ 645us) server_negotiation.cc:299] Negotiation successful
0620 20:15:43.427700 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260620 20:15:43.428980 18151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:43.411538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:42591 (local address 127.16.217.2:56071)
0620 20:15:43.412218 (+ 680us) negotiation.cc:107] Waiting for socket to connect
0620 20:15:43.412255 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 20:15:43.413926 (+ 1671us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:15:43.414628 (+ 702us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:15:43.414636 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:15:43.415126 (+ 490us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:15:43.415661 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:15:43.415677 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:15:43.416684 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:15:43.416690 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:15:43.417927 (+ 1237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:15:43.417938 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:15:43.418735 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:15:43.418742 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:15:43.418970 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:15:43.419719 (+ 749us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:15:43.419754 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:15:43.422366 (+ 2612us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:15:43.425293 (+ 2927us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:15:43.425301 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:15:43.425317 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:15:43.425721 (+ 404us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:15:43.426131 (+ 410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:15:43.426137 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:15:43.426141 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:15:43.426311 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:15:43.426814 (+ 503us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:15:43.426821 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:15:43.427143 (+ 322us) client_negotiation.cc:769] Sending connection context
0620 20:15:43.427400 (+ 257us) client_negotiation.cc:240] Negotiation successful
0620 20:15:43.427906 (+ 506us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":461,"thread_start_us":276,"threads_started":1}
I20260620 20:15:43.430439 18149 heartbeater.cc:347] Connected to a master server at 127.16.217.62:42591
I20260620 20:15:43.430771 18149 heartbeater.cc:464] Registering TS with master...
I20260620 20:15:43.431668 18149 heartbeater.cc:511] Master 127.16.217.62:42591 requested a full tablet report, sending...
I20260620 20:15:43.433334 17826 ts_manager.cc:194] Registered new tserver with Master: 85a41b286f0843c4a57a81fc7bcecf85 (127.16.217.2:39107)
I20260620 20:15:43.434489 17826 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.217.2@KRBTEST.COM'} at 127.16.217.2:56071
WARNING: no policy specified for kudu/127.16.217.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.217.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.217.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.3.keytab.
Entry for principal kudu/127.16.217.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.3.keytab.
I20260620 20:15:43.460770 17252 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
/tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986536687152-17252-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986536687152-17252-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986536687152-17252-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986536687152-17252-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.217.3:0
--local_ip_for_outbound_sockets=127.16.217.3
--webserver_interface=127.16.217.3
--webserver_port=0
--tserver_master_addrs=127.16.217.62:42591
--keytab_file=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.3.keytab
--principal=kudu/127.16.217.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.217.20:45485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986536687152-17252-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986536687152-17252-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986536687152-17252-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:15:43.779182 18154 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:15:43.779489 18154 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:15:43.779590 18154 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:15:43.786855 18154 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:15:43.786950 18154 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:15:43.787060 18154 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.217.3
I20260620 20:15:43.796289 18154 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.217.20:45485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986536687152-17252-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986536687152-17252-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.3.keytab
--principal=kudu/127.16.217.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.217.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986536687152-17252-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.217.3
--webserver_port=0
--tserver_master_addrs=127.16.217.62:42591
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.217.3
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986536687152-17252-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:15:43.797513 18154 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:15:43.798816 18154 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:15:43.811265 18159 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:15:43.811501 18160 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:15:43.811748 18162 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:15:43.812568 18154 server_base.cc:1034] running on GCE node
I20260620 20:15:43.813154 18154 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:15:43.814107 18154 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:15:43.815332 18154 hybrid_clock.cc:648] HybridClock initialized: now 1781986543815281 us; error 41 us; skew 500 ppm
I20260620 20:15:43.815631 18154 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:15:43 dist-test-slave-2hpg krb5kdc[17781](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986543, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.217.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:15:43.818809 18154 init.cc:377] Logged in from keytab as kudu/127.16.217.3@KRBTEST.COM (short username kudu)
I20260620 20:15:43.820281 18154 webserver.cc:460] Webserver started at http://127.16.217.3:42505/ using document root <none> and password file <none>
I20260620 20:15:43.820928 18154 fs_manager.cc:359] Metadata directory not provided
I20260620 20:15:43.821025 18154 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:15:43.821280 18154 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:15:43.824267 18154 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986536687152-17252-0/minicluster-data/ts-2/data/instance:
uuid: "51ea2ce6ea9e44078d860bc3ded9954a"
format_stamp: "Formatted at 2026-06-20 20:15:43 on dist-test-slave-2hpg"
server_key: "e55540256162c90699b6cc087f5ef735"
server_key_iv: "54d8652ad098fb71f4a2757307d936b6"
server_key_version: "encryptionkey@0"
I20260620 20:15:43.825035 18154 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986536687152-17252-0/minicluster-data/ts-2/wal/instance:
uuid: "51ea2ce6ea9e44078d860bc3ded9954a"
format_stamp: "Formatted at 2026-06-20 20:15:43 on dist-test-slave-2hpg"
server_key: "e55540256162c90699b6cc087f5ef735"
server_key_iv: "54d8652ad098fb71f4a2757307d936b6"
server_key_version: "encryptionkey@0"
I20260620 20:15:43.830266 18154 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.003s
I20260620 20:15:43.833608 18169 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:15:43.834956 18154 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 20:15:43.835086 18154 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986536687152-17252-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986536687152-17252-0/minicluster-data/ts-2/wal
uuid: "51ea2ce6ea9e44078d860bc3ded9954a"
format_stamp: "Formatted at 2026-06-20 20:15:43 on dist-test-slave-2hpg"
server_key: "e55540256162c90699b6cc087f5ef735"
server_key_iv: "54d8652ad098fb71f4a2757307d936b6"
server_key_version: "encryptionkey@0"
I20260620 20:15:43.835268 18154 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986536687152-17252-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986536687152-17252-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986536687152-17252-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:15:43.855800 18154 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:15:43.860175 18154 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:15:43.860409 18154 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:15:43.861274 18154 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:15:43.862571 18154 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:15:43.862640 18154 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:15:43.862725 18154 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:15:43.862767 18154 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:15:43.894763 18154 rpc_server.cc:307] RPC server started. Bound to: 127.16.217.3:37819
I20260620 20:15:43.894835 18281 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.217.3:37819 every 8 connection(s)
I20260620 20:15:43.896067 18154 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986536687152-17252-0/minicluster-data/ts-2/data/info.pb
I20260620 20:15:43.896580 17252 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu as pid 18154
I20260620 20:15:43.896787 17252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986536687152-17252-0/minicluster-data/ts-2/wal/instance
I20260620 20:15:43.897324 17252 external_mini_cluster.cc:1428] Setting key cf7f6a0f4b48e32cb39ce6225574dd1f
Jun 20 20:15:43 dist-test-slave-2hpg krb5kdc[17781](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986543, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.217.3@KRBTEST.COM for kudu/127.16.217.62@KRBTEST.COM
I20260620 20:15:43.916639 18017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:43.898754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.217.3:40987 (local address 127.16.217.62:42591)
0620 20:15:43.898938 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0620 20:15:43.898944 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:15:43.900186 (+ 1242us) server_negotiation.cc:407] Connection header received
0620 20:15:43.902054 (+ 1868us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:15:43.902059 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:15:43.902145 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:15:43.902258 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:15:43.903726 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:43.904475 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:43.906111 (+ 1636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:43.906870 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:43.907025 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:15:43.910555 (+ 3530us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:15:43.910560 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:15:43.910563 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:15:43.910592 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:15:43.914127 (+ 3535us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:15:43.914774 (+ 647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:15:43.914778 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:15:43.914780 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:15:43.914885 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:15:43.915230 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:15:43.915233 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:15:43.915235 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:15:43.915489 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:15:43.915589 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0620 20:15:43.916234 (+ 645us) server_negotiation.cc:299] Negotiation successful
0620 20:15:43.916405 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 20:15:43.917549 18284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:43.899308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:42591 (local address 127.16.217.3:40987)
0620 20:15:43.900035 (+ 727us) negotiation.cc:107] Waiting for socket to connect
0620 20:15:43.900072 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 20:15:43.901785 (+ 1713us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:15:43.902490 (+ 705us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:15:43.902498 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:15:43.903017 (+ 519us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:15:43.903551 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:15:43.903567 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:15:43.904611 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:15:43.904617 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:15:43.905982 (+ 1365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:15:43.905994 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:15:43.907146 (+ 1152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:15:43.907152 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:15:43.907386 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:15:43.908043 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:15:43.908069 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:15:43.910374 (+ 2305us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:15:43.914242 (+ 3868us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:15:43.914248 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:15:43.914259 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:15:43.914667 (+ 408us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:15:43.914970 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:15:43.914973 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:15:43.914975 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:15:43.915136 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:15:43.915583 (+ 447us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:15:43.915588 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:15:43.915924 (+ 336us) client_negotiation.cc:769] Sending connection context
0620 20:15:43.916138 (+ 214us) client_negotiation.cc:240] Negotiation successful
0620 20:15:43.916497 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":503,"thread_start_us":333,"threads_started":1}
I20260620 20:15:43.918973 18282 heartbeater.cc:347] Connected to a master server at 127.16.217.62:42591
I20260620 20:15:43.919266 18282 heartbeater.cc:464] Registering TS with master...
I20260620 20:15:43.919950 18282 heartbeater.cc:511] Master 127.16.217.62:42591 requested a full tablet report, sending...
I20260620 20:15:43.921350 17826 ts_manager.cc:194] Registered new tserver with Master: 51ea2ce6ea9e44078d860bc3ded9954a (127.16.217.3:37819)
I20260620 20:15:43.922211 17826 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.217.3@KRBTEST.COM'} at 127.16.217.3:40987
I20260620 20:15:43.925648 17252 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 20:15:43.944748 18017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:43.931665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39052 (local address 127.16.217.62:42591)
0620 20:15:43.931873 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0620 20:15:43.931879 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:15:43.932355 (+ 476us) server_negotiation.cc:407] Connection header received
0620 20:15:43.932514 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:15:43.932518 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:15:43.932574 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:15:43.932653 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:15:43.934565 (+ 1912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:43.935264 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:43.936711 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:43.937108 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:43.937204 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:15:43.939877 (+ 2673us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:15:43.939892 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:15:43.939895 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:15:43.939925 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:15:43.942499 (+ 2574us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:15:43.943095 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:15:43.943099 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:15:43.943102 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:15:43.943217 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:15:43.943537 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:15:43.943540 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:15:43.943542 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:15:43.943818 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:15:43.943994 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0620 20:15:43.944255 (+ 261us) server_negotiation.cc:299] Negotiation successful
0620 20:15:43.944407 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 20:15:43.950840 17826 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39052:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 20:15:43.957653 17826 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 20:15:43.963284 18015 heartbeater.cc:503] Master 127.16.217.62:42591 was elected leader, sending a full tablet report...
I20260620 20:15:44.003260 18294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:43.970462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.2:39107 (local address 127.0.0.1:43932)
0620 20:15:43.991609 (+ 21147us) negotiation.cc:107] Waiting for socket to connect
0620 20:15:43.991626 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 20:15:43.991733 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:15:43.995302 (+ 3569us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:15:43.995307 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:15:43.995337 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:15:43.995614 (+ 277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:15:43.995621 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:15:43.997741 (+ 2120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:15:43.997746 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:15:44.000078 (+ 2332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:15:44.000089 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:15:44.002169 (+ 2080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:15:44.002177 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:15:44.002410 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:15:44.002481 (+ 71us) client_negotiation.cc:769] Sending connection context
0620 20:15:44.002617 (+ 136us) client_negotiation.cc:240] Negotiation successful
0620 20:15:44.002824 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21042,"mutex_wait_us":4,"thread_start_us":25408,"threads_started":1}
I20260620 20:15:44.003753 18298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:43.970928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43932 (local address 127.16.217.2:39107)
0620 20:15:43.994360 (+ 23432us) server_negotiation.cc:206] Beginning negotiation
0620 20:15:43.994372 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 20:15:43.994406 (+ 34us) server_negotiation.cc:407] Connection header received
0620 20:15:43.994520 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:15:43.994527 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:15:43.994818 (+ 291us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:15:43.995026 (+ 208us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:15:43.995772 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:43.997406 (+ 1634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:44.000235 (+ 2829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:44.001762 (+ 1527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:44.001937 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:15:44.003078 (+ 1141us) server_negotiation.cc:1035] Waiting for connection context
0620 20:15:44.003154 (+ 76us) server_negotiation.cc:299] Negotiation successful
0620 20:15:44.003313 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23246,"thread_start_us":23157,"threads_started":1}
I20260620 20:15:44.005483 18297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:43.970112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.1:32937 (local address 127.0.0.1:51648)
0620 20:15:43.990686 (+ 20574us) negotiation.cc:107] Waiting for socket to connect
0620 20:15:43.990758 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0620 20:15:43.991040 (+ 282us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:15:43.997451 (+ 6411us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:15:43.997459 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:15:43.997489 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:15:43.998854 (+ 1365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:15:43.998869 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:15:44.000659 (+ 1790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:15:44.000665 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:15:44.002667 (+ 2002us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:15:44.002680 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:15:44.004599 (+ 1919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:15:44.004605 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:15:44.004785 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:15:44.004825 (+ 40us) client_negotiation.cc:769] Sending connection context
0620 20:15:44.004913 (+ 88us) client_negotiation.cc:240] Negotiation successful
0620 20:15:44.005061 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20216,"mutex_wait_us":49,"thread_start_us":21906,"threads_started":1}
I20260620 20:15:44.005803 18299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:43.971276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57260 (local address 127.16.217.3:37819)
0620 20:15:43.986665 (+ 15389us) server_negotiation.cc:206] Beginning negotiation
0620 20:15:43.986679 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 20:15:43.996289 (+ 9610us) server_negotiation.cc:407] Connection header received
0620 20:15:43.996615 (+ 326us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:15:43.996628 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:15:43.996912 (+ 284us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:15:43.997097 (+ 185us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:15:43.998556 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:44.000253 (+ 1697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:44.002772 (+ 2519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:44.004336 (+ 1564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:44.004485 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:15:44.004698 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0620 20:15:44.005213 (+ 515us) server_negotiation.cc:299] Negotiation successful
0620 20:15:44.005407 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15131,"thread_start_us":15413,"threads_started":1}
I20260620 20:15:44.006551 18296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:43.970880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.3:37819 (local address 127.0.0.1:57260)
0620 20:15:43.996153 (+ 25273us) negotiation.cc:107] Waiting for socket to connect
0620 20:15:43.996170 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 20:15:43.996401 (+ 231us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:15:43.997085 (+ 684us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:15:43.997090 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:15:43.997114 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:15:43.998311 (+ 1197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:15:43.998325 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:15:44.000388 (+ 2063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:15:44.000393 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:15:44.002588 (+ 2195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:15:44.002600 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:15:44.004839 (+ 2239us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:15:44.004846 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:15:44.005009 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:15:44.005041 (+ 32us) client_negotiation.cc:769] Sending connection context
0620 20:15:44.005155 (+ 114us) client_negotiation.cc:240] Negotiation successful
0620 20:15:44.006180 (+ 1025us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25119,"mutex_wait_us":85,"thread_start_us":19234,"threads_started":1}
I20260620 20:15:44.006722 18295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:43.970383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51648 (local address 127.16.217.1:32937)
0620 20:15:43.996934 (+ 26551us) server_negotiation.cc:206] Beginning negotiation
0620 20:15:43.996942 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 20:15:43.996966 (+ 24us) server_negotiation.cc:407] Connection header received
0620 20:15:43.997045 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:15:43.997051 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:15:43.997236 (+ 185us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:15:43.997615 (+ 379us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:15:43.999044 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:44.000530 (+ 1486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:44.002821 (+ 2291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:44.004403 (+ 1582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:44.004552 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:15:44.004770 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0620 20:15:44.004985 (+ 215us) server_negotiation.cc:299] Negotiation successful
0620 20:15:44.006234 (+ 1249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26388,"thread_start_us":26250,"threads_started":1}
I20260620 20:15:44.008455 18084 tablet_service.cc:1467] Processing CreateTablet for tablet 20c9a1304aac4d41a609c704b826107f (DEFAULT_TABLE table=test-table [id=6a06e7b9b4164efd95666015e650f91f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:15:44.009780 18084 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 20c9a1304aac4d41a609c704b826107f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:15:44.010418 17950 tablet_service.cc:1467] Processing CreateTablet for tablet 20c9a1304aac4d41a609c704b826107f (DEFAULT_TABLE table=test-table [id=6a06e7b9b4164efd95666015e650f91f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:15:44.010371 18217 tablet_service.cc:1467] Processing CreateTablet for tablet 20c9a1304aac4d41a609c704b826107f (DEFAULT_TABLE table=test-table [id=6a06e7b9b4164efd95666015e650f91f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:15:44.011718 18217 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 20c9a1304aac4d41a609c704b826107f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:15:44.011868 17950 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 20c9a1304aac4d41a609c704b826107f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:15:44.020717 18300 tablet_bootstrap.cc:492] T 20c9a1304aac4d41a609c704b826107f P 85a41b286f0843c4a57a81fc7bcecf85: Bootstrap starting.
I20260620 20:15:44.024325 18300 tablet_bootstrap.cc:654] T 20c9a1304aac4d41a609c704b826107f P 85a41b286f0843c4a57a81fc7bcecf85: Neither blocks nor log segments found. Creating new log.
I20260620 20:15:44.025312 18300 log.cc:824] T 20c9a1304aac4d41a609c704b826107f P 85a41b286f0843c4a57a81fc7bcecf85: Log is configured to *not* fsync() on all Append() calls
I20260620 20:15:44.027202 18303 tablet_bootstrap.cc:492] T 20c9a1304aac4d41a609c704b826107f P b389338496ea4674878ed902795b3da7: Bootstrap starting.
I20260620 20:15:44.028373 18300 tablet_bootstrap.cc:492] T 20c9a1304aac4d41a609c704b826107f P 85a41b286f0843c4a57a81fc7bcecf85: No bootstrap required, opened a new log
I20260620 20:15:44.028640 18300 ts_tablet_manager.cc:1397] T 20c9a1304aac4d41a609c704b826107f P 85a41b286f0843c4a57a81fc7bcecf85: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.005s
I20260620 20:15:44.030045 18302 tablet_bootstrap.cc:492] T 20c9a1304aac4d41a609c704b826107f P 51ea2ce6ea9e44078d860bc3ded9954a: Bootstrap starting.
I20260620 20:15:44.030735 18303 tablet_bootstrap.cc:654] T 20c9a1304aac4d41a609c704b826107f P b389338496ea4674878ed902795b3da7: Neither blocks nor log segments found. Creating new log.
I20260620 20:15:44.031891 18303 log.cc:824] T 20c9a1304aac4d41a609c704b826107f P b389338496ea4674878ed902795b3da7: Log is configured to *not* fsync() on all Append() calls
I20260620 20:15:44.033735 18302 tablet_bootstrap.cc:654] T 20c9a1304aac4d41a609c704b826107f P 51ea2ce6ea9e44078d860bc3ded9954a: Neither blocks nor log segments found. Creating new log.
I20260620 20:15:44.034657 18302 log.cc:824] T 20c9a1304aac4d41a609c704b826107f P 51ea2ce6ea9e44078d860bc3ded9954a: Log is configured to *not* fsync() on all Append() calls
I20260620 20:15:44.034929 18303 tablet_bootstrap.cc:492] T 20c9a1304aac4d41a609c704b826107f P b389338496ea4674878ed902795b3da7: No bootstrap required, opened a new log
I20260620 20:15:44.035169 18303 ts_tablet_manager.cc:1397] T 20c9a1304aac4d41a609c704b826107f P b389338496ea4674878ed902795b3da7: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.005s
I20260620 20:15:44.036921 18302 tablet_bootstrap.cc:492] T 20c9a1304aac4d41a609c704b826107f P 51ea2ce6ea9e44078d860bc3ded9954a: No bootstrap required, opened a new log
I20260620 20:15:44.037168 18302 ts_tablet_manager.cc:1397] T 20c9a1304aac4d41a609c704b826107f P 51ea2ce6ea9e44078d860bc3ded9954a: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.000s
I20260620 20:15:44.038412 18300 raft_consensus.cc:348] T 20c9a1304aac4d41a609c704b826107f P 85a41b286f0843c4a57a81fc7bcecf85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b389338496ea4674878ed902795b3da7" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 32937 } } peers { permanent_uuid: "85a41b286f0843c4a57a81fc7bcecf85" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 39107 } } peers { permanent_uuid: "51ea2ce6ea9e44078d860bc3ded9954a" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 37819 } }
I20260620 20:15:44.038822 18300 raft_consensus.cc:374] T 20c9a1304aac4d41a609c704b826107f P 85a41b286f0843c4a57a81fc7bcecf85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:15:44.038915 18300 raft_consensus.cc:729] T 20c9a1304aac4d41a609c704b826107f P 85a41b286f0843c4a57a81fc7bcecf85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85a41b286f0843c4a57a81fc7bcecf85, State: Initialized, Role: FOLLOWER
I20260620 20:15:44.039486 18300 consensus_queue.cc:260] T 20c9a1304aac4d41a609c704b826107f P 85a41b286f0843c4a57a81fc7bcecf85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b389338496ea4674878ed902795b3da7" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 32937 } } peers { permanent_uuid: "85a41b286f0843c4a57a81fc7bcecf85" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 39107 } } peers { permanent_uuid: "51ea2ce6ea9e44078d860bc3ded9954a" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 37819 } }
I20260620 20:15:44.040498 18149 heartbeater.cc:503] Master 127.16.217.62:42591 was elected leader, sending a full tablet report...
I20260620 20:15:44.041103 18300 ts_tablet_manager.cc:1428] T 20c9a1304aac4d41a609c704b826107f P 85a41b286f0843c4a57a81fc7bcecf85: Time spent starting tablet: real 0.012s user 0.013s sys 0.000s
I20260620 20:15:44.045115 18303 raft_consensus.cc:348] T 20c9a1304aac4d41a609c704b826107f P b389338496ea4674878ed902795b3da7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b389338496ea4674878ed902795b3da7" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 32937 } } peers { permanent_uuid: "85a41b286f0843c4a57a81fc7bcecf85" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 39107 } } peers { permanent_uuid: "51ea2ce6ea9e44078d860bc3ded9954a" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 37819 } }
I20260620 20:15:44.045517 18303 raft_consensus.cc:374] T 20c9a1304aac4d41a609c704b826107f P b389338496ea4674878ed902795b3da7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:15:44.045614 18303 raft_consensus.cc:729] T 20c9a1304aac4d41a609c704b826107f P b389338496ea4674878ed902795b3da7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b389338496ea4674878ed902795b3da7, State: Initialized, Role: FOLLOWER
I20260620 20:15:44.046232 18303 consensus_queue.cc:260] T 20c9a1304aac4d41a609c704b826107f P b389338496ea4674878ed902795b3da7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b389338496ea4674878ed902795b3da7" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 32937 } } peers { permanent_uuid: "85a41b286f0843c4a57a81fc7bcecf85" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 39107 } } peers { permanent_uuid: "51ea2ce6ea9e44078d860bc3ded9954a" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 37819 } }
I20260620 20:15:44.047477 18302 raft_consensus.cc:348] T 20c9a1304aac4d41a609c704b826107f P 51ea2ce6ea9e44078d860bc3ded9954a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b389338496ea4674878ed902795b3da7" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 32937 } } peers { permanent_uuid: "85a41b286f0843c4a57a81fc7bcecf85" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 39107 } } peers { permanent_uuid: "51ea2ce6ea9e44078d860bc3ded9954a" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 37819 } }
I20260620 20:15:44.047873 18302 raft_consensus.cc:374] T 20c9a1304aac4d41a609c704b826107f P 51ea2ce6ea9e44078d860bc3ded9954a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:15:44.047964 18302 raft_consensus.cc:729] T 20c9a1304aac4d41a609c704b826107f P 51ea2ce6ea9e44078d860bc3ded9954a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51ea2ce6ea9e44078d860bc3ded9954a, State: Initialized, Role: FOLLOWER
I20260620 20:15:44.048127 18303 ts_tablet_manager.cc:1428] T 20c9a1304aac4d41a609c704b826107f P b389338496ea4674878ed902795b3da7: Time spent starting tablet: real 0.013s user 0.010s sys 0.002s
I20260620 20:15:44.048522 18302 consensus_queue.cc:260] T 20c9a1304aac4d41a609c704b826107f P 51ea2ce6ea9e44078d860bc3ded9954a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b389338496ea4674878ed902795b3da7" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 32937 } } peers { permanent_uuid: "85a41b286f0843c4a57a81fc7bcecf85" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 39107 } } peers { permanent_uuid: "51ea2ce6ea9e44078d860bc3ded9954a" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 37819 } }
I20260620 20:15:44.049556 18282 heartbeater.cc:503] Master 127.16.217.62:42591 was elected leader, sending a full tablet report...
I20260620 20:15:44.050269 18302 ts_tablet_manager.cc:1428] T 20c9a1304aac4d41a609c704b826107f P 51ea2ce6ea9e44078d860bc3ded9954a: Time spent starting tablet: real 0.013s user 0.009s sys 0.005s
I20260620 20:15:44.114060 18306 raft_consensus.cc:482] T 20c9a1304aac4d41a609c704b826107f P 85a41b286f0843c4a57a81fc7bcecf85 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 20:15:44.114382 18306 raft_consensus.cc:504] T 20c9a1304aac4d41a609c704b826107f P 85a41b286f0843c4a57a81fc7bcecf85 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b389338496ea4674878ed902795b3da7" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 32937 } } peers { permanent_uuid: "85a41b286f0843c4a57a81fc7bcecf85" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 39107 } } peers { permanent_uuid: "51ea2ce6ea9e44078d860bc3ded9954a" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 37819 } }
I20260620 20:15:44.116061 18306 leader_election.cc:288] T 20c9a1304aac4d41a609c704b826107f P 85a41b286f0843c4a57a81fc7bcecf85 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b389338496ea4674878ed902795b3da7 (127.16.217.1:32937), 51ea2ce6ea9e44078d860bc3ded9954a (127.16.217.3:37819)
I20260620 20:15:44.123452 18299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:44.116578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.217.2:60787 (local address 127.16.217.3:37819)
0620 20:15:44.116810 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0620 20:15:44.116818 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 20:15:44.117509 (+ 691us) server_negotiation.cc:407] Connection header received
0620 20:15:44.117723 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:15:44.117728 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:15:44.117786 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:15:44.117903 (+ 117us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:15:44.118472 (+ 569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:44.119227 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:44.121662 (+ 2435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:44.122487 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:44.122671 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:15:44.122749 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0620 20:15:44.123065 (+ 316us) server_negotiation.cc:299] Negotiation successful
0620 20:15:44.123186 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260620 20:15:44.123456 18295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:44.116446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.217.2:44017 (local address 127.16.217.1:32937)
0620 20:15:44.116673 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0620 20:15:44.116682 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 20:15:44.117518 (+ 836us) server_negotiation.cc:407] Connection header received
0620 20:15:44.117756 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:15:44.117768 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:15:44.117825 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:15:44.117968 (+ 143us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:15:44.118573 (+ 605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:44.119266 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:44.121673 (+ 2407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:44.122505 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:44.122691 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:15:44.122766 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 20:15:44.123065 (+ 299us) server_negotiation.cc:299] Negotiation successful
0620 20:15:44.123189 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 20:15:44.123697 18309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:44.116306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.3:37819 (local address 127.16.217.2:60787)
0620 20:15:44.117394 (+ 1088us) negotiation.cc:107] Waiting for socket to connect
0620 20:15:44.117411 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 20:15:44.117566 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:15:44.117925 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:15:44.117929 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:15:44.117952 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:15:44.118315 (+ 363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:15:44.118326 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:15:44.119366 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:15:44.119374 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:15:44.121493 (+ 2119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:15:44.121503 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:15:44.122706 (+ 1203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:15:44.122715 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:15:44.122907 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:15:44.122930 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 20:15:44.123033 (+ 103us) client_negotiation.cc:240] Negotiation successful
0620 20:15:44.123167 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":924,"mutex_wait_us":130,"thread_start_us":686,"threads_started":1}
I20260620 20:15:44.123785 18310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:44.116397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.1:32937 (local address 127.16.217.2:44017)
0620 20:15:44.117398 (+ 1001us) negotiation.cc:107] Waiting for socket to connect
0620 20:15:44.117416 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 20:15:44.117614 (+ 198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:15:44.117978 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:15:44.117983 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:15:44.118008 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:15:44.118421 (+ 413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:15:44.118429 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:15:44.119415 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:15:44.119424 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:15:44.121492 (+ 2068us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:15:44.121503 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:15:44.122733 (+ 1230us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:15:44.122739 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:15:44.122912 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:15:44.122934 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 20:15:44.123040 (+ 106us) client_negotiation.cc:240] Negotiation successful
0620 20:15:44.123161 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":895,"mutex_wait_us":27,"spinlock_wait_cycles":40192,"thread_start_us":713,"threads_started":1}
I20260620 20:15:44.124780 17970 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20c9a1304aac4d41a609c704b826107f" candidate_uuid: "85a41b286f0843c4a57a81fc7bcecf85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b389338496ea4674878ed902795b3da7" is_pre_election: true
I20260620 20:15:44.124853 18237 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20c9a1304aac4d41a609c704b826107f" candidate_uuid: "85a41b286f0843c4a57a81fc7bcecf85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51ea2ce6ea9e44078d860bc3ded9954a" is_pre_election: true
I20260620 20:15:44.125286 17970 raft_consensus.cc:2454] T 20c9a1304aac4d41a609c704b826107f P b389338496ea4674878ed902795b3da7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85a41b286f0843c4a57a81fc7bcecf85 in term 0.
I20260620 20:15:44.125291 18237 raft_consensus.cc:2454] T 20c9a1304aac4d41a609c704b826107f P 51ea2ce6ea9e44078d860bc3ded9954a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85a41b286f0843c4a57a81fc7bcecf85 in term 0.
I20260620 20:15:44.126000 18037 leader_election.cc:302] T 20c9a1304aac4d41a609c704b826107f P 85a41b286f0843c4a57a81fc7bcecf85 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 51ea2ce6ea9e44078d860bc3ded9954a, 85a41b286f0843c4a57a81fc7bcecf85, b389338496ea4674878ed902795b3da7; no voters:
I20260620 20:15:44.126368 18306 raft_consensus.cc:2781] T 20c9a1304aac4d41a609c704b826107f P 85a41b286f0843c4a57a81fc7bcecf85 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 20:15:44.126511 18306 raft_consensus.cc:482] T 20c9a1304aac4d41a609c704b826107f P 85a41b286f0843c4a57a81fc7bcecf85 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 20:15:44.126624 18306 raft_consensus.cc:3037] T 20c9a1304aac4d41a609c704b826107f P 85a41b286f0843c4a57a81fc7bcecf85 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:15:44.128947 18306 raft_consensus.cc:504] T 20c9a1304aac4d41a609c704b826107f P 85a41b286f0843c4a57a81fc7bcecf85 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b389338496ea4674878ed902795b3da7" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 32937 } } peers { permanent_uuid: "85a41b286f0843c4a57a81fc7bcecf85" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 39107 } } peers { permanent_uuid: "51ea2ce6ea9e44078d860bc3ded9954a" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 37819 } }
I20260620 20:15:44.129482 18306 leader_election.cc:288] T 20c9a1304aac4d41a609c704b826107f P 85a41b286f0843c4a57a81fc7bcecf85 [CANDIDATE]: Term 1 election: Requested vote from peers b389338496ea4674878ed902795b3da7 (127.16.217.1:32937), 51ea2ce6ea9e44078d860bc3ded9954a (127.16.217.3:37819)
I20260620 20:15:44.129942 17970 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20c9a1304aac4d41a609c704b826107f" candidate_uuid: "85a41b286f0843c4a57a81fc7bcecf85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b389338496ea4674878ed902795b3da7"
I20260620 20:15:44.130017 18237 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20c9a1304aac4d41a609c704b826107f" candidate_uuid: "85a41b286f0843c4a57a81fc7bcecf85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51ea2ce6ea9e44078d860bc3ded9954a"
I20260620 20:15:44.130128 17970 raft_consensus.cc:3037] T 20c9a1304aac4d41a609c704b826107f P b389338496ea4674878ed902795b3da7 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:15:44.130247 18237 raft_consensus.cc:3037] T 20c9a1304aac4d41a609c704b826107f P 51ea2ce6ea9e44078d860bc3ded9954a [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:15:44.132599 17970 raft_consensus.cc:2454] T 20c9a1304aac4d41a609c704b826107f P b389338496ea4674878ed902795b3da7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85a41b286f0843c4a57a81fc7bcecf85 in term 1.
I20260620 20:15:44.132710 18237 raft_consensus.cc:2454] T 20c9a1304aac4d41a609c704b826107f P 51ea2ce6ea9e44078d860bc3ded9954a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85a41b286f0843c4a57a81fc7bcecf85 in term 1.
I20260620 20:15:44.133033 18037 leader_election.cc:302] T 20c9a1304aac4d41a609c704b826107f P 85a41b286f0843c4a57a81fc7bcecf85 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a41b286f0843c4a57a81fc7bcecf85, b389338496ea4674878ed902795b3da7; no voters:
I20260620 20:15:44.133291 18306 raft_consensus.cc:2781] T 20c9a1304aac4d41a609c704b826107f P 85a41b286f0843c4a57a81fc7bcecf85 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:15:44.133728 18306 raft_consensus.cc:686] T 20c9a1304aac4d41a609c704b826107f P 85a41b286f0843c4a57a81fc7bcecf85 [term 1 LEADER]: Becoming Leader. State: Replica: 85a41b286f0843c4a57a81fc7bcecf85, State: Running, Role: LEADER
I20260620 20:15:44.134208 18306 consensus_queue.cc:237] T 20c9a1304aac4d41a609c704b826107f P 85a41b286f0843c4a57a81fc7bcecf85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b389338496ea4674878ed902795b3da7" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 32937 } } peers { permanent_uuid: "85a41b286f0843c4a57a81fc7bcecf85" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 39107 } } peers { permanent_uuid: "51ea2ce6ea9e44078d860bc3ded9954a" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 37819 } }
I20260620 20:15:44.138674 17825 catalog_manager.cc:5495] T 20c9a1304aac4d41a609c704b826107f P 85a41b286f0843c4a57a81fc7bcecf85 reported cstate change: term changed from 0 to 1, leader changed from <none> to 85a41b286f0843c4a57a81fc7bcecf85 (127.16.217.2). New cstate: current_term: 1 leader_uuid: "85a41b286f0843c4a57a81fc7bcecf85" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b389338496ea4674878ed902795b3da7" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 32937 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "85a41b286f0843c4a57a81fc7bcecf85" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 39107 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "51ea2ce6ea9e44078d860bc3ded9954a" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 37819 } health_report { overall_health: UNKNOWN } } }
W20260620 20:15:44.147768 18283 tablet.cc:2369] T 20c9a1304aac4d41a609c704b826107f P 51ea2ce6ea9e44078d860bc3ded9954a: Can't schedule compaction. Clean time has not been advanced past its initial value.
Jun 20 20:15:44 dist-test-slave-2hpg krb5kdc[17781](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986541, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.217.1@KRBTEST.COM
W20260620 20:15:44.160312 18150 tablet.cc:2369] T 20c9a1304aac4d41a609c704b826107f P 85a41b286f0843c4a57a81fc7bcecf85: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 20:15:44.160455 18295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:44.146105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51650 (local address 127.16.217.1:32937)
0620 20:15:44.146315 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0620 20:15:44.146323 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 20:15:44.146790 (+ 467us) server_negotiation.cc:407] Connection header received
0620 20:15:44.146917 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:15:44.146921 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:15:44.146986 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:15:44.147065 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:15:44.148877 (+ 1812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:44.149602 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:44.150856 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:44.151272 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:44.151369 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:15:44.155317 (+ 3948us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:15:44.155330 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:15:44.155344 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:15:44.155374 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:15:44.158286 (+ 2912us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:15:44.158865 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:15:44.158871 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:15:44.158874 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:15:44.158985 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:15:44.159278 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:15:44.159281 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:15:44.159283 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:15:44.159622 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:15:44.159824 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0620 20:15:44.160025 (+ 201us) server_negotiation.cc:299] Negotiation successful
0620 20:15:44.160166 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
W20260620 20:15:44.183869 18016 tablet.cc:2369] T 20c9a1304aac4d41a609c704b826107f P b389338496ea4674878ed902795b3da7: Can't schedule compaction. Clean time has not been advanced past its initial value.
Jun 20 20:15:44 dist-test-slave-2hpg krb5kdc[17781](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986544, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 20:15:44 dist-test-slave-2hpg krb5kdc[17781](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986544, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.16.217.1@KRBTEST.COM
I20260620 20:15:44.207206 18295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:44.193137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51662 (local address 127.16.217.1:32937)
0620 20:15:44.193413 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0620 20:15:44.193420 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:15:44.193665 (+ 245us) server_negotiation.cc:407] Connection header received
0620 20:15:44.193956 (+ 291us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:15:44.193961 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:15:44.194028 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:15:44.194117 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:15:44.195534 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:44.196267 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:44.197679 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:44.198179 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:44.198315 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:15:44.202442 (+ 4127us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:15:44.202447 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:15:44.202451 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:15:44.202487 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:15:44.204970 (+ 2483us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:15:44.205487 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:15:44.205490 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:15:44.205492 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:15:44.205606 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:15:44.206011 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:15:44.206014 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:15:44.206016 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:15:44.206323 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:15:44.206585 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0620 20:15:44.206820 (+ 235us) server_negotiation.cc:299] Negotiation successful
0620 20:15:44.206968 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126}
Jun 20 20:15:44 dist-test-slave-2hpg krb5kdc[17781](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986544, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 20:15:44 dist-test-slave-2hpg krb5kdc[17781](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986544, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.217.1@KRBTEST.COM
I20260620 20:15:44.252375 18295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:44.238035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51664 (local address 127.16.217.1:32937)
0620 20:15:44.238278 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0620 20:15:44.238284 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:15:44.238585 (+ 301us) server_negotiation.cc:407] Connection header received
0620 20:15:44.238801 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:15:44.238806 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:15:44.238886 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:15:44.238977 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:15:44.240458 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:44.241148 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:44.242475 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:44.242878 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:44.242983 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:15:44.246678 (+ 3695us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:15:44.246683 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:15:44.246686 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:15:44.246716 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:15:44.249926 (+ 3210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:15:44.250694 (+ 768us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:15:44.250698 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:15:44.250701 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:15:44.250823 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:15:44.251198 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:15:44.251203 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:15:44.251205 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:15:44.251507 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:15:44.251698 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0620 20:15:44.251922 (+ 224us) server_negotiation.cc:299] Negotiation successful
0620 20:15:44.252117 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102}
W20260620 20:15:44.254338 17930 tablet_service.cc:2908] Rejecting scan request for tablet 20c9a1304aac4d41a609c704b826107f: Uninitialized: clean time has not yet been initialized
I20260620 20:15:44.257215 17252 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu pid 17887
I20260620 20:15:44.269336 18295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:44.258405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51676 (local address 127.16.217.1:32937)
0620 20:15:44.258584 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0620 20:15:44.258588 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:15:44.258974 (+ 386us) server_negotiation.cc:407] Connection header received
0620 20:15:44.259097 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:15:44.259101 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:15:44.259167 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:15:44.259245 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:15:44.260309 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:44.260971 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:44.262168 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:44.262588 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:44.262690 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:15:44.264738 (+ 2048us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:15:44.264743 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:15:44.264746 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:15:44.264773 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:15:44.267298 (+ 2525us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:15:44.267870 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:15:44.267874 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:15:44.267876 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:15:44.267988 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:15:44.268282 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:15:44.268284 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:15:44.268287 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:15:44.268565 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:15:44.268788 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0620 20:15:44.268959 (+ 171us) server_negotiation.cc:299] Negotiation successful
0620 20:15:44.269110 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260620 20:15:44.270056 18010 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:15:44.366580 17252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu with pid 17887
I20260620 20:15:44.383477 17252 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu pid 18021
Jun 20 20:15:44 dist-test-slave-2hpg krb5kdc[17781](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986544, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.217.2@KRBTEST.COM
I20260620 20:15:44.397698 18298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:44.384191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43944 (local address 127.16.217.2:39107)
0620 20:15:44.384362 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0620 20:15:44.384367 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:15:44.384387 (+ 20us) server_negotiation.cc:407] Connection header received
0620 20:15:44.384554 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:15:44.384558 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:15:44.384623 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:15:44.384706 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:15:44.385853 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:44.386721 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:44.387969 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:44.388363 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:44.388462 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:15:44.392360 (+ 3898us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:15:44.392373 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:15:44.392389 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:15:44.392416 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:15:44.395202 (+ 2786us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:15:44.395769 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:15:44.395775 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:15:44.395778 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:15:44.395910 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:15:44.396234 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:15:44.396239 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:15:44.396242 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:15:44.396568 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:15:44.396782 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0620 20:15:44.397147 (+ 365us) server_negotiation.cc:299] Negotiation successful
0620 20:15:44.397375 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 20:15:44.398535 18144 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:15:44.506248 17252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu with pid 18021
I20260620 20:15:44.523101 17252 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu pid 18154
Jun 20 20:15:44 dist-test-slave-2hpg krb5kdc[17781](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986544, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.217.3@KRBTEST.COM
I20260620 20:15:44.541234 18299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:44.523886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57272 (local address 127.16.217.3:37819)
0620 20:15:44.524058 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0620 20:15:44.524066 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 20:15:44.524155 (+ 89us) server_negotiation.cc:407] Connection header received
0620 20:15:44.524375 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:15:44.524380 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:15:44.524465 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:15:44.524572 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:15:44.525764 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:44.526551 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:44.527836 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:44.528226 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:44.528356 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:15:44.535714 (+ 7358us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:15:44.535729 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:15:44.535746 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:15:44.535777 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:15:44.538462 (+ 2685us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:15:44.539179 (+ 717us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:15:44.539187 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:15:44.539190 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:15:44.539318 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:15:44.539697 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:15:44.539704 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:15:44.539707 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:15:44.540117 (+ 410us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:15:44.540395 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0620 20:15:44.540714 (+ 319us) server_negotiation.cc:299] Negotiation successful
0620 20:15:44.540920 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 20:15:44.541914 18277 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:15:44.627936 17252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu with pid 18154
I20260620 20:15:44.644413 17252 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu pid 17794
I20260620 20:15:44.645655 17856 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:15:44.751645 17252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu with pid 17794
2026-06-20T20:15:44Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (4089 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986536687152-17252-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 20:15:44 dist-test-slave-2hpg krb5kdc[18330](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 20:15:44 dist-test-slave-2hpg krb5kdc[18330](info): set up 2 sockets
Jun 20 20:15:44 dist-test-slave-2hpg krb5kdc[18330](info): commencing operation
krb5kdc: starting...
W20260620 20:15:45.846136 17252 mini_kdc.cc:121] Time spent starting KDC: real 1.052s user 0.004s sys 0.012s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986536687152-17252-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986536687152-17252-0/krb5kdc/test-user.keytab.
Jun 20 20:15:45 dist-test-slave-2hpg krb5kdc[18330](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986545, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T20:15:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T20:15:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.217.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.217.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.217.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.62.keytab.
Entry for principal kudu/127.16.217.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.62.keytab.
I20260620 20:15:46.031381 17252 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
/tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986536687152-17252-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986536687152-17252-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986536687152-17252-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986536687152-17252-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.217.62:34083
--webserver_interface=127.16.217.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.62.keytab
--principal=kudu/127.16.217.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.217.20:42829
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.217.62:34083
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986536687152-17252-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986536687152-17252-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986536687152-17252-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:15:46.353823 18343 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:15:46.354261 18343 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:15:46.354437 18343 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:15:46.361999 18343 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:15:46.362119 18343 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260620 20:15:46.362192 18343 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:15:46.362319 18343 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:15:46.362417 18343 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:15:46.362548 18343 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:15:46.372972 18343 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.217.20:42829
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986536687152-17252-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986536687152-17252-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.217.62:34083
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.62.keytab
--principal=kudu/127.16.217.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.217.62:34083
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986536687152-17252-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.217.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986536687152-17252-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:15:46.374430 18343 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:15:46.375865 18343 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:15:46.388089 18351 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:15:46.388134 18349 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:15:46.388046 18348 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:15:46.388935 18343 server_base.cc:1034] running on GCE node
I20260620 20:15:46.389784 18343 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:15:46.391296 18343 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:15:46.392570 18343 hybrid_clock.cc:648] HybridClock initialized: now 1781986546392496 us; error 44 us; skew 500 ppm
I20260620 20:15:46.392979 18343 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:15:46 dist-test-slave-2hpg krb5kdc[18330](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986546, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.217.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:15:46.396142 18343 init.cc:377] Logged in from keytab as kudu/127.16.217.62@KRBTEST.COM (short username kudu)
I20260620 20:15:46.397672 18343 webserver.cc:460] Webserver started at http://127.16.217.62:33269/ using document root <none> and password file <none>
I20260620 20:15:46.398459 18343 fs_manager.cc:359] Metadata directory not provided
I20260620 20:15:46.398590 18343 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:15:46.398911 18343 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:15:46.401808 18343 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986536687152-17252-0/minicluster-data/master-0/data/instance:
uuid: "95ec7c2914b94cc7a3ccadd2791d3836"
format_stamp: "Formatted at 2026-06-20 20:15:46 on dist-test-slave-2hpg"
server_key: "a32079c0687d0fe0ee3381931fcdbc5b"
server_key_iv: "bf7ef0ab58d63d0b38182d5b5d8952da"
server_key_version: "encryptionkey@0"
I20260620 20:15:46.402664 18343 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986536687152-17252-0/minicluster-data/master-0/wal/instance:
uuid: "95ec7c2914b94cc7a3ccadd2791d3836"
format_stamp: "Formatted at 2026-06-20 20:15:46 on dist-test-slave-2hpg"
server_key: "a32079c0687d0fe0ee3381931fcdbc5b"
server_key_iv: "bf7ef0ab58d63d0b38182d5b5d8952da"
server_key_version: "encryptionkey@0"
I20260620 20:15:46.407999 18343 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 20:15:46.411633 18358 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:15:46.413097 18343 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 20:15:46.413285 18343 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986536687152-17252-0/minicluster-data/master-0/data,/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986536687152-17252-0/minicluster-data/master-0/wal
uuid: "95ec7c2914b94cc7a3ccadd2791d3836"
format_stamp: "Formatted at 2026-06-20 20:15:46 on dist-test-slave-2hpg"
server_key: "a32079c0687d0fe0ee3381931fcdbc5b"
server_key_iv: "bf7ef0ab58d63d0b38182d5b5d8952da"
server_key_version: "encryptionkey@0"
I20260620 20:15:46.413509 18343 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986536687152-17252-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986536687152-17252-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986536687152-17252-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:15:46.427678 18343 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:15:46.431990 18343 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:15:46.432266 18343 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:15:46.454025 18343 rpc_server.cc:307] RPC server started. Bound to: 127.16.217.62:34083
I20260620 20:15:46.454084 18419 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.217.62:34083 every 8 connection(s)
I20260620 20:15:46.455461 18343 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986536687152-17252-0/minicluster-data/master-0/data/info.pb
I20260620 20:15:46.458664 17252 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu as pid 18343
I20260620 20:15:46.458935 17252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986536687152-17252-0/minicluster-data/master-0/wal/instance
I20260620 20:15:46.459395 18420 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:15:46.459625 17252 external_mini_cluster.cc:1428] Setting key 890a53ea425725cac419abb935e79671
I20260620 20:15:46.474565 18420 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 95ec7c2914b94cc7a3ccadd2791d3836: Bootstrap starting.
Jun 20 20:15:46 dist-test-slave-2hpg krb5kdc[18330](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986545, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.217.62@KRBTEST.COM
I20260620 20:15:46.478497 18420 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 95ec7c2914b94cc7a3ccadd2791d3836: Neither blocks nor log segments found. Creating new log.
I20260620 20:15:46.479588 18420 log.cc:824] T 00000000000000000000000000000000 P 95ec7c2914b94cc7a3ccadd2791d3836: Log is configured to *not* fsync() on all Append() calls
I20260620 20:15:46.482452 18420 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 95ec7c2914b94cc7a3ccadd2791d3836: No bootstrap required, opened a new log
I20260620 20:15:46.484045 18423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:46.461625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51438 (local address 127.16.217.62:34083)
0620 20:15:46.462546 (+ 921us) server_negotiation.cc:206] Beginning negotiation
0620 20:15:46.462561 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 20:15:46.462610 (+ 49us) server_negotiation.cc:407] Connection header received
0620 20:15:46.464996 (+ 2386us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:15:46.465028 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:15:46.465558 (+ 530us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:15:46.466068 (+ 510us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:15:46.467352 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:46.469374 (+ 2022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:46.470705 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:46.471840 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:46.472044 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:15:46.476473 (+ 4429us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:15:46.476485 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:15:46.476500 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:15:46.476542 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:15:46.479941 (+ 3399us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:15:46.480730 (+ 789us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:15:46.480744 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:15:46.480751 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:15:46.480915 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:15:46.481280 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:15:46.481286 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:15:46.481290 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:15:46.481837 (+ 547us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:15:46.482200 (+ 363us) server_negotiation.cc:1035] Waiting for connection context
0620 20:15:46.482629 (+ 429us) server_negotiation.cc:299] Negotiation successful
0620 20:15:46.483039 (+ 410us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":664,"thread_start_us":413,"threads_started":1}
I20260620 20:15:46.490723 18420 raft_consensus.cc:348] T 00000000000000000000000000000000 P 95ec7c2914b94cc7a3ccadd2791d3836 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95ec7c2914b94cc7a3ccadd2791d3836" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 34083 } }
I20260620 20:15:46.491086 18420 raft_consensus.cc:374] T 00000000000000000000000000000000 P 95ec7c2914b94cc7a3ccadd2791d3836 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:15:46.491192 18420 raft_consensus.cc:729] T 00000000000000000000000000000000 P 95ec7c2914b94cc7a3ccadd2791d3836 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 95ec7c2914b94cc7a3ccadd2791d3836, State: Initialized, Role: FOLLOWER
I20260620 20:15:46.491744 18420 consensus_queue.cc:260] T 00000000000000000000000000000000 P 95ec7c2914b94cc7a3ccadd2791d3836 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95ec7c2914b94cc7a3ccadd2791d3836" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 34083 } }
I20260620 20:15:46.491995 18420 raft_consensus.cc:388] T 00000000000000000000000000000000 P 95ec7c2914b94cc7a3ccadd2791d3836 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 20:15:46.492136 18420 raft_consensus.cc:482] T 00000000000000000000000000000000 P 95ec7c2914b94cc7a3ccadd2791d3836 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 20:15:46.492302 18420 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 95ec7c2914b94cc7a3ccadd2791d3836 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:15:46.494757 18420 raft_consensus.cc:504] T 00000000000000000000000000000000 P 95ec7c2914b94cc7a3ccadd2791d3836 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95ec7c2914b94cc7a3ccadd2791d3836" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 34083 } }
I20260620 20:15:46.495203 18420 leader_election.cc:302] T 00000000000000000000000000000000 P 95ec7c2914b94cc7a3ccadd2791d3836 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 95ec7c2914b94cc7a3ccadd2791d3836; no voters:
I20260620 20:15:46.495759 18420 leader_election.cc:288] T 00000000000000000000000000000000 P 95ec7c2914b94cc7a3ccadd2791d3836 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 20:15:46.495935 18425 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 95ec7c2914b94cc7a3ccadd2791d3836 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:15:46.496748 18425 raft_consensus.cc:686] T 00000000000000000000000000000000 P 95ec7c2914b94cc7a3ccadd2791d3836 [term 1 LEADER]: Becoming Leader. State: Replica: 95ec7c2914b94cc7a3ccadd2791d3836, State: Running, Role: LEADER
I20260620 20:15:46.497306 18425 consensus_queue.cc:237] T 00000000000000000000000000000000 P 95ec7c2914b94cc7a3ccadd2791d3836 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95ec7c2914b94cc7a3ccadd2791d3836" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 34083 } }
I20260620 20:15:46.498000 18420 sys_catalog.cc:564] T 00000000000000000000000000000000 P 95ec7c2914b94cc7a3ccadd2791d3836 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:15:46.500782 18427 sys_catalog.cc:455] T 00000000000000000000000000000000 P 95ec7c2914b94cc7a3ccadd2791d3836 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 95ec7c2914b94cc7a3ccadd2791d3836. Latest consensus state: current_term: 1 leader_uuid: "95ec7c2914b94cc7a3ccadd2791d3836" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95ec7c2914b94cc7a3ccadd2791d3836" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 34083 } } }
I20260620 20:15:46.501185 18427 sys_catalog.cc:458] T 00000000000000000000000000000000 P 95ec7c2914b94cc7a3ccadd2791d3836 [sys.catalog]: This master's current role is: LEADER
I20260620 20:15:46.504465 18426 sys_catalog.cc:455] T 00000000000000000000000000000000 P 95ec7c2914b94cc7a3ccadd2791d3836 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "95ec7c2914b94cc7a3ccadd2791d3836" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95ec7c2914b94cc7a3ccadd2791d3836" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 34083 } } }
I20260620 20:15:46.504747 18426 sys_catalog.cc:458] T 00000000000000000000000000000000 P 95ec7c2914b94cc7a3ccadd2791d3836 [sys.catalog]: This master's current role is: LEADER
W20260620 20:15:46.508599 18441 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 95ec7c2914b94cc7a3ccadd2791d3836: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 20:15:46.508760 18441 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 20:15:46.508976 18436 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 20:15:46.510798 18436 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 20:15:46.519735 18436 catalog_manager.cc:1347] Generated new cluster ID: a7abdd47b5324412b6feebba9325cd2c
I20260620 20:15:46.519868 18436 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 20:15:46.564114 18436 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 20:15:46.565490 18436 catalog_manager.cc:1504] Loading token signing keys...
I20260620 20:15:46.579886 18436 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 95ec7c2914b94cc7a3ccadd2791d3836: Generated new TSK 0
I20260620 20:15:46.580720 18436 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.217.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.217.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.217.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.1.keytab.
Entry for principal kudu/127.16.217.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.1.keytab.
I20260620 20:15:46.630724 17252 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
/tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986536687152-17252-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986536687152-17252-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986536687152-17252-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986536687152-17252-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.217.1:0
--local_ip_for_outbound_sockets=127.16.217.1
--webserver_interface=127.16.217.1
--webserver_port=0
--tserver_master_addrs=127.16.217.62:34083
--keytab_file=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.1.keytab
--principal=kudu/127.16.217.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.217.20:42829
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986536687152-17252-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986536687152-17252-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986536687152-17252-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:15:46.951129 18447 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:15:46.951469 18447 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:15:46.951650 18447 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:15:46.958750 18447 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:15:46.958909 18447 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:15:46.959074 18447 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 20:15:46.959218 18447 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.217.1
I20260620 20:15:46.968776 18447 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.217.20:42829
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986536687152-17252-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986536687152-17252-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.1.keytab
--principal=kudu/127.16.217.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.217.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986536687152-17252-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.217.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.16.217.62:34083
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.217.1
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986536687152-17252-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:15:46.970162 18447 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:15:46.971531 18447 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:15:46.984607 18455 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:15:46.984647 18453 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:15:46.985134 18452 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:15:46.986512 18447 server_base.cc:1034] running on GCE node
I20260620 20:15:46.987217 18447 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:15:46.988116 18447 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:15:46.989368 18447 hybrid_clock.cc:648] HybridClock initialized: now 1781986546989275 us; error 70 us; skew 500 ppm
I20260620 20:15:46.989696 18447 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:15:46 dist-test-slave-2hpg krb5kdc[18330](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986546, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.217.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:15:46.992882 18447 init.cc:377] Logged in from keytab as kudu/127.16.217.1@KRBTEST.COM (short username kudu)
I20260620 20:15:46.994429 18447 webserver.cc:460] Webserver started at http://127.16.217.1:37633/ using document root <none> and password file <none>
I20260620 20:15:46.995113 18447 fs_manager.cc:359] Metadata directory not provided
I20260620 20:15:46.995198 18447 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:15:46.995455 18447 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:15:46.998457 18447 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986536687152-17252-0/minicluster-data/ts-0/data/instance:
uuid: "eaf918de62a94ff88d3d82f3fbaf1b61"
format_stamp: "Formatted at 2026-06-20 20:15:46 on dist-test-slave-2hpg"
server_key: "0e9aeb3f57a264cbfbb293edb6c9357e"
server_key_iv: "c4a575e7e6438215f1298abdcf953a12"
server_key_version: "encryptionkey@0"
I20260620 20:15:46.999238 18447 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986536687152-17252-0/minicluster-data/ts-0/wal/instance:
uuid: "eaf918de62a94ff88d3d82f3fbaf1b61"
format_stamp: "Formatted at 2026-06-20 20:15:46 on dist-test-slave-2hpg"
server_key: "0e9aeb3f57a264cbfbb293edb6c9357e"
server_key_iv: "c4a575e7e6438215f1298abdcf953a12"
server_key_version: "encryptionkey@0"
I20260620 20:15:47.004436 18447 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 20:15:47.008157 18462 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:15:47.009608 18447 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 20:15:47.009806 18447 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986536687152-17252-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986536687152-17252-0/minicluster-data/ts-0/wal
uuid: "eaf918de62a94ff88d3d82f3fbaf1b61"
format_stamp: "Formatted at 2026-06-20 20:15:46 on dist-test-slave-2hpg"
server_key: "0e9aeb3f57a264cbfbb293edb6c9357e"
server_key_iv: "c4a575e7e6438215f1298abdcf953a12"
server_key_version: "encryptionkey@0"
I20260620 20:15:47.010170 18447 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986536687152-17252-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986536687152-17252-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986536687152-17252-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:15:47.029543 18447 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:15:47.034062 18447 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:15:47.034345 18447 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:15:47.037192 18447 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:15:47.037300 18447 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:15:47.037456 18447 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:15:47.037612 18447 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
Jun 20 20:15:47 dist-test-slave-2hpg krb5kdc[18330](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986546, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.217.1@KRBTEST.COM for kudu/127.16.217.62@KRBTEST.COM
I20260620 20:15:47.062549 18471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:47.037707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.217.1:51193 (local address 127.16.217.62:34083)
0620 20:15:47.038807 (+ 1100us) server_negotiation.cc:206] Beginning negotiation
0620 20:15:47.038817 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 20:15:47.039440 (+ 623us) server_negotiation.cc:407] Connection header received
0620 20:15:47.041617 (+ 2177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:15:47.041624 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:15:47.041722 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:15:47.041855 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:15:47.044284 (+ 2429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:47.045627 (+ 1343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:47.048998 (+ 3371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:47.049591 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:47.049740 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:15:47.055404 (+ 5664us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:15:47.055410 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:15:47.055413 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:15:47.055447 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:15:47.058347 (+ 2900us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:15:47.059583 (+ 1236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:15:47.059588 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:15:47.059591 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:15:47.059752 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:15:47.060501 (+ 749us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:15:47.060507 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:15:47.060513 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:15:47.060819 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:15:47.061011 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0620 20:15:47.061939 (+ 928us) server_negotiation.cc:299] Negotiation successful
0620 20:15:47.062157 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":945,"thread_start_us":826,"threads_started":1}
I20260620 20:15:47.064121 18472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:47.038403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:34083 (local address 127.16.217.1:51193)
0620 20:15:47.039258 (+ 855us) negotiation.cc:107] Waiting for socket to connect
0620 20:15:47.039300 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 20:15:47.041311 (+ 2011us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:15:47.042202 (+ 891us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:15:47.042221 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:15:47.043118 (+ 897us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:15:47.044027 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:15:47.044079 (+ 52us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:15:47.045916 (+ 1837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:15:47.045922 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:15:47.048820 (+ 2898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:15:47.048842 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:15:47.049953 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:15:47.049961 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:15:47.050343 (+ 382us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:15:47.051247 (+ 904us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:15:47.051269 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:15:47.055098 (+ 3829us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:15:47.058561 (+ 3463us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:15:47.058575 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:15:47.058601 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:15:47.059376 (+ 775us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:15:47.059911 (+ 535us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:15:47.059918 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:15:47.059928 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:15:47.060337 (+ 409us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:15:47.061096 (+ 759us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:15:47.061110 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:15:47.061540 (+ 430us) client_negotiation.cc:769] Sending connection context
0620 20:15:47.061831 (+ 291us) client_negotiation.cc:240] Negotiation successful
0620 20:15:47.062436 (+ 605us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":626,"thread_start_us":404,"threads_started":1}
I20260620 20:15:47.086761 18551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:47.072109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:34083 (local address 127.16.217.1:40309)
0620 20:15:47.072698 (+ 589us) negotiation.cc:107] Waiting for socket to connect
0620 20:15:47.072721 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 20:15:47.072898 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:15:47.073315 (+ 417us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:15:47.073319 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:15:47.073811 (+ 492us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:15:47.074362 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:15:47.074373 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:15:47.075708 (+ 1335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:15:47.075714 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:15:47.077076 (+ 1362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:15:47.077088 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:15:47.078086 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:15:47.078096 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:15:47.078368 (+ 272us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:15:47.079295 (+ 927us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:15:47.079327 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:15:47.080664 (+ 1337us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:15:47.084297 (+ 3633us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:15:47.084304 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:15:47.084308 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:15:47.084695 (+ 387us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:15:47.085066 (+ 371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:15:47.085071 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:15:47.085075 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:15:47.085198 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:15:47.085689 (+ 491us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:15:47.085694 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:15:47.085931 (+ 237us) client_negotiation.cc:769] Sending connection context
0620 20:15:47.086141 (+ 210us) client_negotiation.cc:240] Negotiation successful
0620 20:15:47.086435 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":445,"thread_start_us":301,"threads_started":1}
I20260620 20:15:47.087007 18471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:47.072233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.217.1:40309 (local address 127.16.217.62:34083)
0620 20:15:47.072415 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0620 20:15:47.072419 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:15:47.072836 (+ 417us) server_negotiation.cc:407] Connection header received
0620 20:15:47.073052 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:15:47.073059 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:15:47.073146 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:15:47.073285 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:15:47.074508 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:47.075582 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:47.077228 (+ 1646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:47.077841 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:47.078018 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:15:47.080863 (+ 2845us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:15:47.080868 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:15:47.080871 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:15:47.080900 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:15:47.084161 (+ 3261us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:15:47.084814 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:15:47.084822 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:15:47.084828 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:15:47.084958 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:15:47.085305 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:15:47.085320 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:15:47.085323 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:15:47.085574 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:15:47.085725 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0620 20:15:47.086257 (+ 532us) server_negotiation.cc:299] Negotiation successful
0620 20:15:47.086574 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 20:15:47.092887 18447 rpc_server.cc:307] RPC server started. Bound to: 127.16.217.1:43825
I20260620 20:15:47.092903 18582 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.217.1:43825 every 8 connection(s)
I20260620 20:15:47.094130 18447 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986536687152-17252-0/minicluster-data/ts-0/data/info.pb
I20260620 20:15:47.096426 18583 heartbeater.cc:347] Connected to a master server at 127.16.217.62:34083
I20260620 20:15:47.096652 18583 heartbeater.cc:464] Registering TS with master...
I20260620 20:15:47.097278 18583 heartbeater.cc:511] Master 127.16.217.62:34083 requested a full tablet report, sending...
I20260620 20:15:47.099372 18375 ts_manager.cc:194] Registered new tserver with Master: eaf918de62a94ff88d3d82f3fbaf1b61 (127.16.217.1:43825)
I20260620 20:15:47.099872 17252 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu as pid 18447
I20260620 20:15:47.100086 17252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986536687152-17252-0/minicluster-data/ts-0/wal/instance
I20260620 20:15:47.100891 17252 external_mini_cluster.cc:1428] Setting key 24b0c1157d884ee1d198b9c79ce31f54
I20260620 20:15:47.101481 18375 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.217.1@KRBTEST.COM'} at 127.16.217.1:51193
WARNING: no policy specified for kudu/127.16.217.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.217.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.217.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.2.keytab.
Entry for principal kudu/127.16.217.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.2.keytab.
I20260620 20:15:47.140399 17252 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
/tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986536687152-17252-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986536687152-17252-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986536687152-17252-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986536687152-17252-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.217.2:0
--local_ip_for_outbound_sockets=127.16.217.2
--webserver_interface=127.16.217.2
--webserver_port=0
--tserver_master_addrs=127.16.217.62:34083
--keytab_file=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.2.keytab
--principal=kudu/127.16.217.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.217.20:42829
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986536687152-17252-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986536687152-17252-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986536687152-17252-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:15:47.460767 18587 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:15:47.461185 18587 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:15:47.461351 18587 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:15:47.473326 18587 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:15:47.473492 18587 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:15:47.473651 18587 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 20:15:47.473783 18587 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.217.2
I20260620 20:15:47.485260 18587 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.217.20:42829
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986536687152-17252-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986536687152-17252-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.2.keytab
--principal=kudu/127.16.217.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.217.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986536687152-17252-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.217.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.16.217.62:34083
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.217.2
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986536687152-17252-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:15:47.486661 18587 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:15:47.488055 18587 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:15:47.502038 18592 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:15:47.502074 18593 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:15:47.502125 18595 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:15:47.503178 18587 server_base.cc:1034] running on GCE node
I20260620 20:15:47.503779 18587 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:15:47.504745 18587 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:15:47.506011 18587 hybrid_clock.cc:648] HybridClock initialized: now 1781986547505944 us; error 34 us; skew 500 ppm
I20260620 20:15:47.506393 18587 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:15:47 dist-test-slave-2hpg krb5kdc[18330](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986547, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.217.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:15:47.509461 18587 init.cc:377] Logged in from keytab as kudu/127.16.217.2@KRBTEST.COM (short username kudu)
I20260620 20:15:47.510861 18587 webserver.cc:460] Webserver started at http://127.16.217.2:37321/ using document root <none> and password file <none>
I20260620 20:15:47.511646 18587 fs_manager.cc:359] Metadata directory not provided
I20260620 20:15:47.511737 18587 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:15:47.511997 18587 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:15:47.514827 18587 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986536687152-17252-0/minicluster-data/ts-1/data/instance:
uuid: "03c1ea73e1504bf9b87e1ac2bb358219"
format_stamp: "Formatted at 2026-06-20 20:15:47 on dist-test-slave-2hpg"
server_key: "58ecefe276df3ecd05686939388e5a3e"
server_key_iv: "7fa106ddc3dc26017ffafdba95f1576f"
server_key_version: "encryptionkey@0"
I20260620 20:15:47.515544 18587 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986536687152-17252-0/minicluster-data/ts-1/wal/instance:
uuid: "03c1ea73e1504bf9b87e1ac2bb358219"
format_stamp: "Formatted at 2026-06-20 20:15:47 on dist-test-slave-2hpg"
server_key: "58ecefe276df3ecd05686939388e5a3e"
server_key_iv: "7fa106ddc3dc26017ffafdba95f1576f"
server_key_version: "encryptionkey@0"
I20260620 20:15:47.520798 18587 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260620 20:15:47.524209 18602 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:15:47.525480 18587 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260620 20:15:47.525642 18587 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986536687152-17252-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986536687152-17252-0/minicluster-data/ts-1/wal
uuid: "03c1ea73e1504bf9b87e1ac2bb358219"
format_stamp: "Formatted at 2026-06-20 20:15:47 on dist-test-slave-2hpg"
server_key: "58ecefe276df3ecd05686939388e5a3e"
server_key_iv: "7fa106ddc3dc26017ffafdba95f1576f"
server_key_version: "encryptionkey@0"
I20260620 20:15:47.525851 18587 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986536687152-17252-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986536687152-17252-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986536687152-17252-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:15:47.548869 18587 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:15:47.553313 18587 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:15:47.553633 18587 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:15:47.556408 18587 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:15:47.556533 18587 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:15:47.556718 18587 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:15:47.556821 18587 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 20:15:47 dist-test-slave-2hpg krb5kdc[18330](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986547, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.217.2@KRBTEST.COM for kudu/127.16.217.62@KRBTEST.COM
I20260620 20:15:47.580528 18471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:47.557108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.217.2:51933 (local address 127.16.217.62:34083)
0620 20:15:47.557329 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0620 20:15:47.557336 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:15:47.558927 (+ 1591us) server_negotiation.cc:407] Connection header received
0620 20:15:47.562186 (+ 3259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:15:47.562192 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:15:47.562288 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:15:47.562413 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:15:47.564736 (+ 2323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:47.565711 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:47.567944 (+ 2233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:47.568724 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:47.568940 (+ 216us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:15:47.573332 (+ 4392us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:15:47.573340 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:15:47.573345 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:15:47.573391 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:15:47.577058 (+ 3667us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:15:47.578083 (+ 1025us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:15:47.578088 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:15:47.578092 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:15:47.578229 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:15:47.578669 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:15:47.578674 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:15:47.578679 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:15:47.578964 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:15:47.579120 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0620 20:15:47.579962 (+ 842us) server_negotiation.cc:299] Negotiation successful
0620 20:15:47.580170 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260620 20:15:47.581525 18611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:47.557709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:34083 (local address 127.16.217.2:51933)
0620 20:15:47.558688 (+ 979us) negotiation.cc:107] Waiting for socket to connect
0620 20:15:47.558736 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0620 20:15:47.561663 (+ 2927us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:15:47.562845 (+ 1182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:15:47.562860 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:15:47.563669 (+ 809us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:15:47.564455 (+ 786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:15:47.564481 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:15:47.565894 (+ 1413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:15:47.565900 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:15:47.567758 (+ 1858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:15:47.567776 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:15:47.569089 (+ 1313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:15:47.569099 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:15:47.569448 (+ 349us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:15:47.570282 (+ 834us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:15:47.570328 (+ 46us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:15:47.573076 (+ 2748us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:15:47.577197 (+ 4121us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:15:47.577206 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:15:47.577227 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:15:47.577945 (+ 718us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:15:47.578338 (+ 393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:15:47.578344 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:15:47.578347 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:15:47.578547 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:15:47.579093 (+ 546us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:15:47.579101 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:15:47.579524 (+ 423us) client_negotiation.cc:769] Sending connection context
0620 20:15:47.579837 (+ 313us) client_negotiation.cc:240] Negotiation successful
0620 20:15:47.580341 (+ 504us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":649,"thread_start_us":401,"threads_started":1}
I20260620 20:15:47.604243 18471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:47.587534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.217.2:40099 (local address 127.16.217.62:34083)
0620 20:15:47.587704 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0620 20:15:47.587708 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:15:47.588422 (+ 714us) server_negotiation.cc:407] Connection header received
0620 20:15:47.588637 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:15:47.588644 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:15:47.588708 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:15:47.588809 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:15:47.590277 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:47.591565 (+ 1288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:47.594313 (+ 2748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:47.594863 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:47.595033 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:15:47.597606 (+ 2573us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:15:47.597612 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:15:47.597620 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:15:47.597662 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:15:47.600949 (+ 3287us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:15:47.602108 (+ 1159us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:15:47.602112 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:15:47.602114 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:15:47.602237 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:15:47.602744 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:15:47.602748 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:15:47.602750 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:15:47.603112 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:15:47.603228 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0620 20:15:47.603852 (+ 624us) server_negotiation.cc:299] Negotiation successful
0620 20:15:47.604016 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 20:15:47.604948 18666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:47.587720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:34083 (local address 127.16.217.2:40099)
0620 20:15:47.588319 (+ 599us) negotiation.cc:107] Waiting for socket to connect
0620 20:15:47.588333 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 20:15:47.588501 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:15:47.588834 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:15:47.588839 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:15:47.589541 (+ 702us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:15:47.590091 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:15:47.590110 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:15:47.591825 (+ 1715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:15:47.591835 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:15:47.594161 (+ 2326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:15:47.594179 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:15:47.595095 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:15:47.595103 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:15:47.595294 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:15:47.596146 (+ 852us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:15:47.596163 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:15:47.597353 (+ 1190us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:15:47.601084 (+ 3731us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:15:47.601094 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:15:47.601102 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:15:47.601963 (+ 861us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:15:47.602355 (+ 392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:15:47.602363 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:15:47.602370 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:15:47.602582 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:15:47.603241 (+ 659us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:15:47.603249 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:15:47.603527 (+ 278us) client_negotiation.cc:769] Sending connection context
0620 20:15:47.603721 (+ 194us) client_negotiation.cc:240] Negotiation successful
0620 20:15:47.604188 (+ 467us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":481,"thread_start_us":351,"threads_started":1}
I20260620 20:15:47.615571 18587 rpc_server.cc:307] RPC server started. Bound to: 127.16.217.2:42015
I20260620 20:15:47.615597 18721 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.217.2:42015 every 8 connection(s)
I20260620 20:15:47.616966 18587 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986536687152-17252-0/minicluster-data/ts-1/data/info.pb
I20260620 20:15:47.617457 17252 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu as pid 18587
I20260620 20:15:47.617687 17252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986536687152-17252-0/minicluster-data/ts-1/wal/instance
I20260620 20:15:47.618271 17252 external_mini_cluster.cc:1428] Setting key 72c6c5c85cf514e72f42431312a47014
I20260620 20:15:47.619618 18722 heartbeater.cc:347] Connected to a master server at 127.16.217.62:34083
I20260620 20:15:47.620002 18722 heartbeater.cc:464] Registering TS with master...
I20260620 20:15:47.621101 18722 heartbeater.cc:511] Master 127.16.217.62:34083 requested a full tablet report, sending...
I20260620 20:15:47.622996 18375 ts_manager.cc:194] Registered new tserver with Master: 03c1ea73e1504bf9b87e1ac2bb358219 (127.16.217.2:42015)
I20260620 20:15:47.624223 18375 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.217.2@KRBTEST.COM'} at 127.16.217.2:51933
WARNING: no policy specified for kudu/127.16.217.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.217.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.217.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.3.keytab.
Entry for principal kudu/127.16.217.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.3.keytab.
I20260620 20:15:47.659238 17252 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
/tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986536687152-17252-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986536687152-17252-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986536687152-17252-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986536687152-17252-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.217.3:0
--local_ip_for_outbound_sockets=127.16.217.3
--webserver_interface=127.16.217.3
--webserver_port=0
--tserver_master_addrs=127.16.217.62:34083
--keytab_file=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.3.keytab
--principal=kudu/127.16.217.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.217.20:42829
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986536687152-17252-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986536687152-17252-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986536687152-17252-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:15:47.977880 18726 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:15:47.978171 18726 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:15:47.978266 18726 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:15:47.985808 18726 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:15:47.985966 18726 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:15:47.986065 18726 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 20:15:47.986146 18726 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.217.3
I20260620 20:15:47.995672 18726 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.217.20:42829
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986536687152-17252-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986536687152-17252-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.3.keytab
--principal=kudu/127.16.217.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.217.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986536687152-17252-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.217.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.16.217.62:34083
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.217.3
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986536687152-17252-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:15:47.996865 18726 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:15:47.998229 18726 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:15:48.011467 18731 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:15:48.011488 18732 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:15:48.012104 18726 server_base.cc:1034] running on GCE node
W20260620 20:15:48.012205 18734 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:15:48.012902 18726 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:15:48.013962 18726 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:15:48.015275 18726 hybrid_clock.cc:648] HybridClock initialized: now 1781986548015185 us; error 38 us; skew 500 ppm
I20260620 20:15:48.015636 18726 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:15:48 dist-test-slave-2hpg krb5kdc[18330](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986548, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.217.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:15:48.018991 18726 init.cc:377] Logged in from keytab as kudu/127.16.217.3@KRBTEST.COM (short username kudu)
I20260620 20:15:48.020557 18726 webserver.cc:460] Webserver started at http://127.16.217.3:45203/ using document root <none> and password file <none>
I20260620 20:15:48.021252 18726 fs_manager.cc:359] Metadata directory not provided
I20260620 20:15:48.021385 18726 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:15:48.021736 18726 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:15:48.024741 18726 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986536687152-17252-0/minicluster-data/ts-2/data/instance:
uuid: "f6c53b45956f4822b1c5e81eefe212ae"
format_stamp: "Formatted at 2026-06-20 20:15:48 on dist-test-slave-2hpg"
server_key: "547a55017d8cfda6b90779cde129876e"
server_key_iv: "7cb4d973292fc6b80c4e2d45e5f6500a"
server_key_version: "encryptionkey@0"
I20260620 20:15:48.025533 18726 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986536687152-17252-0/minicluster-data/ts-2/wal/instance:
uuid: "f6c53b45956f4822b1c5e81eefe212ae"
format_stamp: "Formatted at 2026-06-20 20:15:48 on dist-test-slave-2hpg"
server_key: "547a55017d8cfda6b90779cde129876e"
server_key_iv: "7cb4d973292fc6b80c4e2d45e5f6500a"
server_key_version: "encryptionkey@0"
I20260620 20:15:48.030867 18726 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.001s
I20260620 20:15:48.034348 18741 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:15:48.035686 18726 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260620 20:15:48.035851 18726 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986536687152-17252-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986536687152-17252-0/minicluster-data/ts-2/wal
uuid: "f6c53b45956f4822b1c5e81eefe212ae"
format_stamp: "Formatted at 2026-06-20 20:15:48 on dist-test-slave-2hpg"
server_key: "547a55017d8cfda6b90779cde129876e"
server_key_iv: "7cb4d973292fc6b80c4e2d45e5f6500a"
server_key_version: "encryptionkey@0"
I20260620 20:15:48.036060 18726 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986536687152-17252-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986536687152-17252-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986536687152-17252-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:15:48.062209 18726 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:15:48.066470 18726 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:15:48.066753 18726 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:15:48.069500 18726 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:15:48.069615 18726 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:15:48.069785 18726 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:15:48.069892 18726 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 20:15:48 dist-test-slave-2hpg krb5kdc[18330](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986548, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.217.3@KRBTEST.COM for kudu/127.16.217.62@KRBTEST.COM
I20260620 20:15:48.096112 18750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:48.070939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:34083 (local address 127.16.217.3:47125)
0620 20:15:48.071795 (+ 856us) negotiation.cc:107] Waiting for socket to connect
0620 20:15:48.071859 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0620 20:15:48.073975 (+ 2116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:15:48.074958 (+ 983us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:15:48.074976 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:15:48.075728 (+ 752us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:15:48.076547 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:15:48.076581 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:15:48.078334 (+ 1753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:15:48.078339 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:15:48.080361 (+ 2022us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:15:48.080380 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:15:48.081602 (+ 1222us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:15:48.081616 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:15:48.081936 (+ 320us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:15:48.082824 (+ 888us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:15:48.082867 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:15:48.086103 (+ 3236us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:15:48.090624 (+ 4521us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:15:48.090638 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:15:48.090659 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:15:48.091430 (+ 771us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:15:48.091960 (+ 530us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:15:48.091968 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:15:48.091972 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:15:48.092259 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:15:48.092998 (+ 739us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:15:48.093005 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:15:48.093508 (+ 503us) client_negotiation.cc:769] Sending connection context
0620 20:15:48.093902 (+ 394us) client_negotiation.cc:240] Negotiation successful
0620 20:15:48.094488 (+ 586us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":582,"thread_start_us":375,"threads_started":1}
I20260620 20:15:48.097299 18471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:48.070298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.217.3:47125 (local address 127.16.217.62:34083)
0620 20:15:48.070491 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0620 20:15:48.070498 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:15:48.072001 (+ 1503us) server_negotiation.cc:407] Connection header received
0620 20:15:48.074389 (+ 2388us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:15:48.074398 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:15:48.074508 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:15:48.074630 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:15:48.076754 (+ 2124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:48.078121 (+ 1367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:48.080540 (+ 2419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:48.081319 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:48.081486 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:15:48.086349 (+ 4863us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:15:48.086362 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:15:48.086366 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:15:48.086402 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:15:48.090438 (+ 4036us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:15:48.091578 (+ 1140us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:15:48.091587 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:15:48.091590 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:15:48.091795 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:15:48.092404 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:15:48.092409 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:15:48.092411 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:15:48.092772 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:15:48.092926 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0620 20:15:48.096654 (+ 3728us) server_negotiation.cc:299] Negotiation successful
0620 20:15:48.096945 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 20:15:48.104271 18583 heartbeater.cc:503] Master 127.16.217.62:34083 was elected leader, sending a full tablet report...
I20260620 20:15:48.122152 18726 rpc_server.cc:307] RPC server started. Bound to: 127.16.217.3:42419
I20260620 20:15:48.122197 18860 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.217.3:42419 every 8 connection(s)
I20260620 20:15:48.122979 18843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:48.109355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:34083 (local address 127.16.217.3:41277)
0620 20:15:48.110074 (+ 719us) negotiation.cc:107] Waiting for socket to connect
0620 20:15:48.110099 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0620 20:15:48.110250 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:15:48.110595 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:15:48.110600 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:15:48.111065 (+ 465us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:15:48.111342 (+ 277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:15:48.111350 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:15:48.112349 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:15:48.112355 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:15:48.113444 (+ 1089us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:15:48.113455 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:15:48.114464 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:15:48.114471 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:15:48.114632 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:15:48.115408 (+ 776us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:15:48.115437 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:15:48.116219 (+ 782us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:15:48.120004 (+ 3785us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:15:48.120010 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:15:48.120013 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:15:48.120560 (+ 547us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:15:48.120964 (+ 404us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:15:48.120969 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:15:48.120973 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:15:48.121143 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:15:48.121927 (+ 784us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:15:48.121934 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:15:48.122170 (+ 236us) client_negotiation.cc:769] Sending connection context
0620 20:15:48.122343 (+ 173us) client_negotiation.cc:240] Negotiation successful
0620 20:15:48.122645 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":564,"thread_start_us":387,"threads_started":1}
I20260620 20:15:48.123587 18726 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986536687152-17252-0/minicluster-data/ts-2/data/info.pb
I20260620 20:15:48.123747 18471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:48.109183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.217.3:41277 (local address 127.16.217.62:34083)
0620 20:15:48.109361 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0620 20:15:48.109367 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:15:48.110183 (+ 816us) server_negotiation.cc:407] Connection header received
0620 20:15:48.110379 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:15:48.110384 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:15:48.110467 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:15:48.110569 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:15:48.111492 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:48.112209 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:48.113589 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:48.114225 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:48.114389 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:15:48.116394 (+ 2005us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:15:48.116400 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:15:48.116405 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:15:48.116444 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:15:48.119766 (+ 3322us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:15:48.120677 (+ 911us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:15:48.120682 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:15:48.120684 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:15:48.120817 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:15:48.121327 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:15:48.121332 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:15:48.121335 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:15:48.121763 (+ 428us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:15:48.121991 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0620 20:15:48.123294 (+ 1303us) server_negotiation.cc:299] Negotiation successful
0620 20:15:48.123487 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 20:15:48.125774 18861 heartbeater.cc:347] Connected to a master server at 127.16.217.62:34083
I20260620 20:15:48.126173 17252 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu as pid 18726
I20260620 20:15:48.126216 18861 heartbeater.cc:464] Registering TS with master...
I20260620 20:15:48.126533 17252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986536687152-17252-0/minicluster-data/ts-2/wal/instance
I20260620 20:15:48.127111 17252 external_mini_cluster.cc:1428] Setting key 7e507f2b57a6d78c932d53e7cb03ad44
I20260620 20:15:48.127159 18861 heartbeater.cc:511] Master 127.16.217.62:34083 requested a full tablet report, sending...
I20260620 20:15:48.128659 18375 ts_manager.cc:194] Registered new tserver with Master: f6c53b45956f4822b1c5e81eefe212ae (127.16.217.3:42419)
I20260620 20:15:48.129645 18375 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.217.3@KRBTEST.COM'} at 127.16.217.3:47125
I20260620 20:15:48.130373 17252 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 20:15:48 dist-test-slave-2hpg krb5kdc[18330](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986548, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 20:15:48 dist-test-slave-2hpg krb5kdc[18330](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986548, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.16.217.62@KRBTEST.COM
I20260620 20:15:48.175243 18471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:48.161627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51448 (local address 127.16.217.62:34083)
0620 20:15:48.161808 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0620 20:15:48.161812 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:15:48.162131 (+ 319us) server_negotiation.cc:407] Connection header received
0620 20:15:48.162326 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:15:48.162331 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:15:48.162407 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:15:48.162503 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:15:48.163811 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:48.164594 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:48.165825 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:48.166317 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:48.166426 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:15:48.170396 (+ 3970us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:15:48.170402 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:15:48.170405 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:15:48.170439 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:15:48.172960 (+ 2521us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:15:48.173574 (+ 614us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:15:48.173577 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:15:48.173580 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:15:48.173698 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:15:48.174118 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:15:48.174121 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:15:48.174123 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:15:48.174385 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:15:48.174560 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0620 20:15:48.174812 (+ 252us) server_negotiation.cc:299] Negotiation successful
0620 20:15:48.174998 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260620 20:15:48.179499 18375 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51448:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 20:15:48.187146 18375 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 20:15:48.209772 18876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:48.199953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48548 (local address 127.16.217.2:42015)
0620 20:15:48.200619 (+ 666us) server_negotiation.cc:206] Beginning negotiation
0620 20:15:48.200629 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 20:15:48.201324 (+ 695us) server_negotiation.cc:407] Connection header received
0620 20:15:48.201702 (+ 378us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:15:48.201713 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:15:48.202024 (+ 311us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:15:48.202209 (+ 185us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:15:48.203272 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:48.204721 (+ 1449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:48.207759 (+ 3038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:48.208615 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:48.208780 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:15:48.208988 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0620 20:15:48.209276 (+ 288us) server_negotiation.cc:299] Negotiation successful
0620 20:15:48.209449 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":495,"thread_start_us":1255,"threads_started":1}
I20260620 20:15:48.210021 18878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:48.199364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.2:42015 (local address 127.0.0.1:48548)
0620 20:15:48.201184 (+ 1820us) negotiation.cc:107] Waiting for socket to connect
0620 20:15:48.201203 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 20:15:48.201467 (+ 264us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:15:48.202565 (+ 1098us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:15:48.202570 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:15:48.202598 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:15:48.203067 (+ 469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:15:48.203080 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:15:48.204866 (+ 1786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:15:48.204869 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:15:48.207571 (+ 2702us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:15:48.207584 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:15:48.208807 (+ 1223us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:15:48.208813 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:15:48.209006 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:15:48.209056 (+ 50us) client_negotiation.cc:769] Sending connection context
0620 20:15:48.209219 (+ 163us) client_negotiation.cc:240] Negotiation successful
0620 20:15:48.209414 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1695,"mutex_wait_us":584,"thread_start_us":2143,"threads_started":1}
I20260620 20:15:48.211027 18877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:48.200409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60226 (local address 127.16.217.3:42419)
0620 20:15:48.201213 (+ 804us) server_negotiation.cc:206] Beginning negotiation
0620 20:15:48.201224 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 20:15:48.201264 (+ 40us) server_negotiation.cc:407] Connection header received
0620 20:15:48.201369 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:15:48.201378 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:15:48.201680 (+ 302us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:15:48.201922 (+ 242us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:15:48.202508 (+ 586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:48.204416 (+ 1908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:48.207668 (+ 3252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:48.209284 (+ 1616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:48.209417 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:15:48.209647 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0620 20:15:48.210562 (+ 915us) server_negotiation.cc:299] Negotiation successful
0620 20:15:48.210711 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":601,"thread_start_us":950,"threads_started":1}
I20260620 20:15:48.211967 18874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:48.199344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.3:42419 (local address 127.0.0.1:60226)
0620 20:15:48.200178 (+ 834us) negotiation.cc:107] Waiting for socket to connect
0620 20:15:48.200222 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 20:15:48.200387 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:15:48.201920 (+ 1533us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:15:48.201927 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:15:48.201966 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:15:48.202309 (+ 343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:15:48.202319 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:15:48.204579 (+ 2260us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:15:48.204588 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:15:48.207496 (+ 2908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:15:48.207512 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:15:48.209848 (+ 2336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:15:48.209857 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:15:48.210114 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:15:48.210149 (+ 35us) client_negotiation.cc:769] Sending connection context
0620 20:15:48.210258 (+ 109us) client_negotiation.cc:240] Negotiation successful
0620 20:15:48.211619 (+ 1361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":584,"mutex_wait_us":136,"thread_start_us":444,"threads_started":1}
I20260620 20:15:48.214300 18875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:48.200624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.1:43825 (local address 127.0.0.1:51432)
0620 20:15:48.201427 (+ 803us) negotiation.cc:107] Waiting for socket to connect
0620 20:15:48.201444 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 20:15:48.201563 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:15:48.206436 (+ 4873us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:15:48.206442 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:15:48.206475 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:15:48.206905 (+ 430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:15:48.206920 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:15:48.209043 (+ 2123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:15:48.209048 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:15:48.211427 (+ 2379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:15:48.211442 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:15:48.213459 (+ 2017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:15:48.213466 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:15:48.213647 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:15:48.213684 (+ 37us) client_negotiation.cc:769] Sending connection context
0620 20:15:48.213793 (+ 109us) client_negotiation.cc:240] Negotiation successful
0620 20:15:48.213965 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":660,"thread_start_us":254,"threads_started":1}
I20260620 20:15:48.214310 18879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:48.201066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51432 (local address 127.16.217.1:43825)
0620 20:15:48.205791 (+ 4725us) server_negotiation.cc:206] Beginning negotiation
0620 20:15:48.205799 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 20:15:48.205831 (+ 32us) server_negotiation.cc:407] Connection header received
0620 20:15:48.205953 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:15:48.205960 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:15:48.206176 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:15:48.207124 (+ 948us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:15:48.207325 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:48.208874 (+ 1549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:48.211662 (+ 2788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:48.213242 (+ 1580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:48.213416 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:15:48.213669 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0620 20:15:48.213831 (+ 162us) server_negotiation.cc:299] Negotiation successful
0620 20:15:48.214009 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4565,"thread_start_us":4423,"threads_started":1}
I20260620 20:15:48.213792 18652 tablet_service.cc:1467] Processing CreateTablet for tablet 85250d317c554a5cb5a4942faa24290c (DEFAULT_TABLE table=test-table [id=679ba587a14e469c9ce91fd6a705d0f4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:15:48.215237 18652 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 85250d317c554a5cb5a4942faa24290c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:15:48.215508 18791 tablet_service.cc:1467] Processing CreateTablet for tablet 85250d317c554a5cb5a4942faa24290c (DEFAULT_TABLE table=test-table [id=679ba587a14e469c9ce91fd6a705d0f4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:15:48.216902 18791 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 85250d317c554a5cb5a4942faa24290c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:15:48.217757 18513 tablet_service.cc:1467] Processing CreateTablet for tablet 85250d317c554a5cb5a4942faa24290c (DEFAULT_TABLE table=test-table [id=679ba587a14e469c9ce91fd6a705d0f4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:15:48.218997 18513 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 85250d317c554a5cb5a4942faa24290c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:15:48.227319 18880 tablet_bootstrap.cc:492] T 85250d317c554a5cb5a4942faa24290c P 03c1ea73e1504bf9b87e1ac2bb358219: Bootstrap starting.
I20260620 20:15:48.230988 18880 tablet_bootstrap.cc:654] T 85250d317c554a5cb5a4942faa24290c P 03c1ea73e1504bf9b87e1ac2bb358219: Neither blocks nor log segments found. Creating new log.
I20260620 20:15:48.231252 18881 tablet_bootstrap.cc:492] T 85250d317c554a5cb5a4942faa24290c P f6c53b45956f4822b1c5e81eefe212ae: Bootstrap starting.
I20260620 20:15:48.232733 18880 log.cc:824] T 85250d317c554a5cb5a4942faa24290c P 03c1ea73e1504bf9b87e1ac2bb358219: Log is configured to *not* fsync() on all Append() calls
I20260620 20:15:48.233224 18882 tablet_bootstrap.cc:492] T 85250d317c554a5cb5a4942faa24290c P eaf918de62a94ff88d3d82f3fbaf1b61: Bootstrap starting.
I20260620 20:15:48.235095 18881 tablet_bootstrap.cc:654] T 85250d317c554a5cb5a4942faa24290c P f6c53b45956f4822b1c5e81eefe212ae: Neither blocks nor log segments found. Creating new log.
I20260620 20:15:48.236418 18881 log.cc:824] T 85250d317c554a5cb5a4942faa24290c P f6c53b45956f4822b1c5e81eefe212ae: Log is configured to *not* fsync() on all Append() calls
I20260620 20:15:48.236570 18882 tablet_bootstrap.cc:654] T 85250d317c554a5cb5a4942faa24290c P eaf918de62a94ff88d3d82f3fbaf1b61: Neither blocks nor log segments found. Creating new log.
I20260620 20:15:48.237134 18880 tablet_bootstrap.cc:492] T 85250d317c554a5cb5a4942faa24290c P 03c1ea73e1504bf9b87e1ac2bb358219: No bootstrap required, opened a new log
I20260620 20:15:48.237404 18880 ts_tablet_manager.cc:1397] T 85250d317c554a5cb5a4942faa24290c P 03c1ea73e1504bf9b87e1ac2bb358219: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.001s
I20260620 20:15:48.237699 18882 log.cc:824] T 85250d317c554a5cb5a4942faa24290c P eaf918de62a94ff88d3d82f3fbaf1b61: Log is configured to *not* fsync() on all Append() calls
I20260620 20:15:48.239746 18881 tablet_bootstrap.cc:492] T 85250d317c554a5cb5a4942faa24290c P f6c53b45956f4822b1c5e81eefe212ae: No bootstrap required, opened a new log
I20260620 20:15:48.240170 18881 ts_tablet_manager.cc:1397] T 85250d317c554a5cb5a4942faa24290c P f6c53b45956f4822b1c5e81eefe212ae: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.002s
I20260620 20:15:48.245277 18880 raft_consensus.cc:348] T 85250d317c554a5cb5a4942faa24290c P 03c1ea73e1504bf9b87e1ac2bb358219 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03c1ea73e1504bf9b87e1ac2bb358219" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42015 } } peers { permanent_uuid: "f6c53b45956f4822b1c5e81eefe212ae" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42419 } } peers { permanent_uuid: "eaf918de62a94ff88d3d82f3fbaf1b61" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 43825 } }
I20260620 20:15:48.245651 18880 raft_consensus.cc:374] T 85250d317c554a5cb5a4942faa24290c P 03c1ea73e1504bf9b87e1ac2bb358219 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:15:48.245801 18880 raft_consensus.cc:729] T 85250d317c554a5cb5a4942faa24290c P 03c1ea73e1504bf9b87e1ac2bb358219 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03c1ea73e1504bf9b87e1ac2bb358219, State: Initialized, Role: FOLLOWER
I20260620 20:15:48.246592 18880 consensus_queue.cc:260] T 85250d317c554a5cb5a4942faa24290c P 03c1ea73e1504bf9b87e1ac2bb358219 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03c1ea73e1504bf9b87e1ac2bb358219" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42015 } } peers { permanent_uuid: "f6c53b45956f4822b1c5e81eefe212ae" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42419 } } peers { permanent_uuid: "eaf918de62a94ff88d3d82f3fbaf1b61" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 43825 } }
I20260620 20:15:48.247675 18722 heartbeater.cc:503] Master 127.16.217.62:34083 was elected leader, sending a full tablet report...
I20260620 20:15:48.248529 18882 tablet_bootstrap.cc:492] T 85250d317c554a5cb5a4942faa24290c P eaf918de62a94ff88d3d82f3fbaf1b61: No bootstrap required, opened a new log
I20260620 20:15:48.248833 18882 ts_tablet_manager.cc:1397] T 85250d317c554a5cb5a4942faa24290c P eaf918de62a94ff88d3d82f3fbaf1b61: Time spent bootstrapping tablet: real 0.016s user 0.011s sys 0.004s
I20260620 20:15:48.250258 18880 ts_tablet_manager.cc:1428] T 85250d317c554a5cb5a4942faa24290c P 03c1ea73e1504bf9b87e1ac2bb358219: Time spent starting tablet: real 0.013s user 0.006s sys 0.004s
I20260620 20:15:48.251770 18881 raft_consensus.cc:348] T 85250d317c554a5cb5a4942faa24290c P f6c53b45956f4822b1c5e81eefe212ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03c1ea73e1504bf9b87e1ac2bb358219" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42015 } } peers { permanent_uuid: "f6c53b45956f4822b1c5e81eefe212ae" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42419 } } peers { permanent_uuid: "eaf918de62a94ff88d3d82f3fbaf1b61" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 43825 } }
I20260620 20:15:48.252615 18881 raft_consensus.cc:374] T 85250d317c554a5cb5a4942faa24290c P f6c53b45956f4822b1c5e81eefe212ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:15:48.252719 18881 raft_consensus.cc:729] T 85250d317c554a5cb5a4942faa24290c P f6c53b45956f4822b1c5e81eefe212ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6c53b45956f4822b1c5e81eefe212ae, State: Initialized, Role: FOLLOWER
I20260620 20:15:48.253414 18881 consensus_queue.cc:260] T 85250d317c554a5cb5a4942faa24290c P f6c53b45956f4822b1c5e81eefe212ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03c1ea73e1504bf9b87e1ac2bb358219" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42015 } } peers { permanent_uuid: "f6c53b45956f4822b1c5e81eefe212ae" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42419 } } peers { permanent_uuid: "eaf918de62a94ff88d3d82f3fbaf1b61" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 43825 } }
I20260620 20:15:48.254546 18861 heartbeater.cc:503] Master 127.16.217.62:34083 was elected leader, sending a full tablet report...
I20260620 20:15:48.255149 18881 ts_tablet_manager.cc:1428] T 85250d317c554a5cb5a4942faa24290c P f6c53b45956f4822b1c5e81eefe212ae: Time spent starting tablet: real 0.015s user 0.011s sys 0.003s
I20260620 20:15:48.258580 18882 raft_consensus.cc:348] T 85250d317c554a5cb5a4942faa24290c P eaf918de62a94ff88d3d82f3fbaf1b61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03c1ea73e1504bf9b87e1ac2bb358219" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42015 } } peers { permanent_uuid: "f6c53b45956f4822b1c5e81eefe212ae" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42419 } } peers { permanent_uuid: "eaf918de62a94ff88d3d82f3fbaf1b61" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 43825 } }
I20260620 20:15:48.258956 18882 raft_consensus.cc:374] T 85250d317c554a5cb5a4942faa24290c P eaf918de62a94ff88d3d82f3fbaf1b61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:15:48.259030 18882 raft_consensus.cc:729] T 85250d317c554a5cb5a4942faa24290c P eaf918de62a94ff88d3d82f3fbaf1b61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eaf918de62a94ff88d3d82f3fbaf1b61, State: Initialized, Role: FOLLOWER
I20260620 20:15:48.259536 18882 consensus_queue.cc:260] T 85250d317c554a5cb5a4942faa24290c P eaf918de62a94ff88d3d82f3fbaf1b61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03c1ea73e1504bf9b87e1ac2bb358219" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42015 } } peers { permanent_uuid: "f6c53b45956f4822b1c5e81eefe212ae" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42419 } } peers { permanent_uuid: "eaf918de62a94ff88d3d82f3fbaf1b61" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 43825 } }
I20260620 20:15:48.261034 18882 ts_tablet_manager.cc:1428] T 85250d317c554a5cb5a4942faa24290c P eaf918de62a94ff88d3d82f3fbaf1b61: Time spent starting tablet: real 0.012s user 0.011s sys 0.000s
W20260620 20:15:48.346551 18584 tablet.cc:2369] T 85250d317c554a5cb5a4942faa24290c P eaf918de62a94ff88d3d82f3fbaf1b61: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 20:15:48.373258 18723 tablet.cc:2369] T 85250d317c554a5cb5a4942faa24290c P 03c1ea73e1504bf9b87e1ac2bb358219: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 20:15:48.374192 18888 raft_consensus.cc:482] T 85250d317c554a5cb5a4942faa24290c P eaf918de62a94ff88d3d82f3fbaf1b61 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 20:15:48.374521 18888 raft_consensus.cc:504] T 85250d317c554a5cb5a4942faa24290c P eaf918de62a94ff88d3d82f3fbaf1b61 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03c1ea73e1504bf9b87e1ac2bb358219" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42015 } } peers { permanent_uuid: "f6c53b45956f4822b1c5e81eefe212ae" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42419 } } peers { permanent_uuid: "eaf918de62a94ff88d3d82f3fbaf1b61" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 43825 } }
W20260620 20:15:48.375236 18862 tablet.cc:2369] T 85250d317c554a5cb5a4942faa24290c P f6c53b45956f4822b1c5e81eefe212ae: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 20:15:48.376233 18888 leader_election.cc:288] T 85250d317c554a5cb5a4942faa24290c P eaf918de62a94ff88d3d82f3fbaf1b61 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 03c1ea73e1504bf9b87e1ac2bb358219 (127.16.217.2:42015), f6c53b45956f4822b1c5e81eefe212ae (127.16.217.3:42419)
I20260620 20:15:48.383392 18876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:48.376556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.217.1:56625 (local address 127.16.217.2:42015)
0620 20:15:48.377166 (+ 610us) server_negotiation.cc:206] Beginning negotiation
0620 20:15:48.377172 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:15:48.378171 (+ 999us) server_negotiation.cc:407] Connection header received
0620 20:15:48.378366 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:15:48.378370 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:15:48.378439 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:15:48.378530 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:15:48.379191 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:48.379930 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:48.381616 (+ 1686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:48.382419 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:48.382600 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:15:48.382673 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 20:15:48.382967 (+ 294us) server_negotiation.cc:299] Negotiation successful
0620 20:15:48.383085 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":466}
I20260620 20:15:48.383481 18890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:48.377164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.2:42015 (local address 127.16.217.1:56625)
0620 20:15:48.378092 (+ 928us) negotiation.cc:107] Waiting for socket to connect
0620 20:15:48.378106 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 20:15:48.378213 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:15:48.378552 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:15:48.378556 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:15:48.378580 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:15:48.379027 (+ 447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:15:48.379035 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:15:48.380062 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:15:48.380071 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:15:48.381461 (+ 1390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:15:48.381472 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:15:48.382616 (+ 1144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:15:48.382621 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:15:48.382774 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:15:48.382833 (+ 59us) client_negotiation.cc:769] Sending connection context
0620 20:15:48.382944 (+ 111us) client_negotiation.cc:240] Negotiation successful
0620 20:15:48.383080 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":822,"mutex_wait_us":8,"thread_start_us":490,"threads_started":1}
I20260620 20:15:48.383543 18877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:48.376771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.217.1:54975 (local address 127.16.217.3:42419)
0620 20:15:48.376953 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0620 20:15:48.376957 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:15:48.378109 (+ 1152us) server_negotiation.cc:407] Connection header received
0620 20:15:48.378344 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:15:48.378347 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:15:48.378406 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:15:48.378504 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:15:48.379128 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:48.379815 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:48.381807 (+ 1992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:48.382621 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:48.382805 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:15:48.382885 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0620 20:15:48.383165 (+ 280us) server_negotiation.cc:299] Negotiation successful
0620 20:15:48.383279 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 20:15:48.384014 18889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:48.377143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.3:42419 (local address 127.16.217.1:54975)
0620 20:15:48.377996 (+ 853us) negotiation.cc:107] Waiting for socket to connect
0620 20:15:48.378012 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 20:15:48.378187 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:15:48.378529 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:15:48.378532 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:15:48.378555 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:15:48.378989 (+ 434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:15:48.379001 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:15:48.379940 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:15:48.379948 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:15:48.381649 (+ 1701us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:15:48.381668 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:15:48.382839 (+ 1171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:15:48.382845 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:15:48.382993 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:15:48.383014 (+ 21us) client_negotiation.cc:769] Sending connection context
0620 20:15:48.383124 (+ 110us) client_negotiation.cc:240] Negotiation successful
0620 20:15:48.383284 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":675,"mutex_wait_us":36,"thread_start_us":480,"threads_started":1}
I20260620 20:15:48.384730 18677 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85250d317c554a5cb5a4942faa24290c" candidate_uuid: "eaf918de62a94ff88d3d82f3fbaf1b61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "03c1ea73e1504bf9b87e1ac2bb358219" is_pre_election: true
I20260620 20:15:48.384999 18811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85250d317c554a5cb5a4942faa24290c" candidate_uuid: "eaf918de62a94ff88d3d82f3fbaf1b61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6c53b45956f4822b1c5e81eefe212ae" is_pre_election: true
I20260620 20:15:48.385221 18677 raft_consensus.cc:2454] T 85250d317c554a5cb5a4942faa24290c P 03c1ea73e1504bf9b87e1ac2bb358219 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eaf918de62a94ff88d3d82f3fbaf1b61 in term 0.
I20260620 20:15:48.385517 18811 raft_consensus.cc:2454] T 85250d317c554a5cb5a4942faa24290c P f6c53b45956f4822b1c5e81eefe212ae [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eaf918de62a94ff88d3d82f3fbaf1b61 in term 0.
I20260620 20:15:48.385972 18465 leader_election.cc:302] T 85250d317c554a5cb5a4942faa24290c P eaf918de62a94ff88d3d82f3fbaf1b61 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 03c1ea73e1504bf9b87e1ac2bb358219, eaf918de62a94ff88d3d82f3fbaf1b61; no voters:
I20260620 20:15:48.386868 18888 raft_consensus.cc:2781] T 85250d317c554a5cb5a4942faa24290c P eaf918de62a94ff88d3d82f3fbaf1b61 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 20:15:48.387020 18888 raft_consensus.cc:482] T 85250d317c554a5cb5a4942faa24290c P eaf918de62a94ff88d3d82f3fbaf1b61 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 20:15:48.387156 18888 raft_consensus.cc:3037] T 85250d317c554a5cb5a4942faa24290c P eaf918de62a94ff88d3d82f3fbaf1b61 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:15:48.389776 18888 raft_consensus.cc:504] T 85250d317c554a5cb5a4942faa24290c P eaf918de62a94ff88d3d82f3fbaf1b61 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03c1ea73e1504bf9b87e1ac2bb358219" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42015 } } peers { permanent_uuid: "f6c53b45956f4822b1c5e81eefe212ae" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42419 } } peers { permanent_uuid: "eaf918de62a94ff88d3d82f3fbaf1b61" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 43825 } }
I20260620 20:15:48.390475 18888 leader_election.cc:288] T 85250d317c554a5cb5a4942faa24290c P eaf918de62a94ff88d3d82f3fbaf1b61 [CANDIDATE]: Term 1 election: Requested vote from peers 03c1ea73e1504bf9b87e1ac2bb358219 (127.16.217.2:42015), f6c53b45956f4822b1c5e81eefe212ae (127.16.217.3:42419)
I20260620 20:15:48.390942 18811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85250d317c554a5cb5a4942faa24290c" candidate_uuid: "eaf918de62a94ff88d3d82f3fbaf1b61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6c53b45956f4822b1c5e81eefe212ae"
I20260620 20:15:48.390981 18677 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85250d317c554a5cb5a4942faa24290c" candidate_uuid: "eaf918de62a94ff88d3d82f3fbaf1b61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "03c1ea73e1504bf9b87e1ac2bb358219"
I20260620 20:15:48.391136 18811 raft_consensus.cc:3037] T 85250d317c554a5cb5a4942faa24290c P f6c53b45956f4822b1c5e81eefe212ae [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:15:48.391175 18677 raft_consensus.cc:3037] T 85250d317c554a5cb5a4942faa24290c P 03c1ea73e1504bf9b87e1ac2bb358219 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:15:48.393236 18811 raft_consensus.cc:2454] T 85250d317c554a5cb5a4942faa24290c P f6c53b45956f4822b1c5e81eefe212ae [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eaf918de62a94ff88d3d82f3fbaf1b61 in term 1.
I20260620 20:15:48.393352 18677 raft_consensus.cc:2454] T 85250d317c554a5cb5a4942faa24290c P 03c1ea73e1504bf9b87e1ac2bb358219 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eaf918de62a94ff88d3d82f3fbaf1b61 in term 1.
I20260620 20:15:48.393675 18463 leader_election.cc:302] T 85250d317c554a5cb5a4942faa24290c P eaf918de62a94ff88d3d82f3fbaf1b61 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: eaf918de62a94ff88d3d82f3fbaf1b61, f6c53b45956f4822b1c5e81eefe212ae; no voters:
I20260620 20:15:48.394052 18888 raft_consensus.cc:2781] T 85250d317c554a5cb5a4942faa24290c P eaf918de62a94ff88d3d82f3fbaf1b61 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:15:48.394529 18888 raft_consensus.cc:686] T 85250d317c554a5cb5a4942faa24290c P eaf918de62a94ff88d3d82f3fbaf1b61 [term 1 LEADER]: Becoming Leader. State: Replica: eaf918de62a94ff88d3d82f3fbaf1b61, State: Running, Role: LEADER
I20260620 20:15:48.394979 18888 consensus_queue.cc:237] T 85250d317c554a5cb5a4942faa24290c P eaf918de62a94ff88d3d82f3fbaf1b61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03c1ea73e1504bf9b87e1ac2bb358219" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42015 } } peers { permanent_uuid: "f6c53b45956f4822b1c5e81eefe212ae" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42419 } } peers { permanent_uuid: "eaf918de62a94ff88d3d82f3fbaf1b61" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 43825 } }
I20260620 20:15:48.399691 18374 catalog_manager.cc:5495] T 85250d317c554a5cb5a4942faa24290c P eaf918de62a94ff88d3d82f3fbaf1b61 reported cstate change: term changed from 0 to 1, leader changed from <none> to eaf918de62a94ff88d3d82f3fbaf1b61 (127.16.217.1). New cstate: current_term: 1 leader_uuid: "eaf918de62a94ff88d3d82f3fbaf1b61" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03c1ea73e1504bf9b87e1ac2bb358219" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42015 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f6c53b45956f4822b1c5e81eefe212ae" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42419 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eaf918de62a94ff88d3d82f3fbaf1b61" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 43825 } health_report { overall_health: HEALTHY } } }
I20260620 20:15:48.453495 18879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:48.448154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51444 (local address 127.16.217.1:43825)
0620 20:15:48.448406 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0620 20:15:48.448414 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 20:15:48.448490 (+ 76us) server_negotiation.cc:407] Connection header received
0620 20:15:48.448765 (+ 275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:15:48.448768 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:15:48.448850 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:15:48.448999 (+ 149us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 20:15:48.449333 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:48.450116 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:48.451394 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:48.451793 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:48.451902 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:15:48.452384 (+ 482us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 20:15:48.452898 (+ 514us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 20:15:48.453004 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0620 20:15:48.453125 (+ 121us) server_negotiation.cc:299] Negotiation successful
0620 20:15:48.453185 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 20:15:48.461741 18677 raft_consensus.cc:1261] T 85250d317c554a5cb5a4942faa24290c P 03c1ea73e1504bf9b87e1ac2bb358219 [term 1 FOLLOWER]: Refusing update from remote peer eaf918de62a94ff88d3d82f3fbaf1b61: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:15:48.461921 18811 raft_consensus.cc:1261] T 85250d317c554a5cb5a4942faa24290c P f6c53b45956f4822b1c5e81eefe212ae [term 1 FOLLOWER]: Refusing update from remote peer eaf918de62a94ff88d3d82f3fbaf1b61: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:15:48.462782 18891 consensus_queue.cc:1035] T 85250d317c554a5cb5a4942faa24290c P eaf918de62a94ff88d3d82f3fbaf1b61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6c53b45956f4822b1c5e81eefe212ae" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42419 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:15:48.463171 18888 consensus_queue.cc:1035] T 85250d317c554a5cb5a4942faa24290c P eaf918de62a94ff88d3d82f3fbaf1b61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "03c1ea73e1504bf9b87e1ac2bb358219" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42015 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:15:48.485656 18894 mvcc.cc:205] Tried to move back new op lower bound from 7299016902489137152 to 7299016902229331968. Current Snapshot: MvccSnapshot[applied={T|T < 7299016902489137152 or (T in {7299016902489137152})}]
I20260620 20:15:48.491173 18896 mvcc.cc:205] Tried to move back new op lower bound from 7299016902489137152 to 7299016902229331968. Current Snapshot: MvccSnapshot[applied={T|T < 7299016902489137152}]
I20260620 20:15:48.505105 18374 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51448:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 20:15:48.505438 18374 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51448:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 20:15:48.508869 18374 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 95ec7c2914b94cc7a3ccadd2791d3836: Sending DeleteTablet for 3 replicas of tablet 85250d317c554a5cb5a4942faa24290c
I20260620 20:15:48.510011 18652 tablet_service.cc:1514] Processing DeleteTablet for tablet 85250d317c554a5cb5a4942faa24290c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:15:48 UTC) from {username='kudu', principal='kudu/127.16.217.62@KRBTEST.COM'} at 127.0.0.1:48548
I20260620 20:15:48.510232 18513 tablet_service.cc:1514] Processing DeleteTablet for tablet 85250d317c554a5cb5a4942faa24290c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:15:48 UTC) from {username='kudu', principal='kudu/127.16.217.62@KRBTEST.COM'} at 127.0.0.1:51432
I20260620 20:15:48.510303 18791 tablet_service.cc:1514] Processing DeleteTablet for tablet 85250d317c554a5cb5a4942faa24290c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:15:48 UTC) from {username='kudu', principal='kudu/127.16.217.62@KRBTEST.COM'} at 127.0.0.1:60226
I20260620 20:15:48.512374 18375 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51448:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 20:15:48.513195 18375 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 20:15:48.521934 18906 tablet_replica.cc:330] T 85250d317c554a5cb5a4942faa24290c P f6c53b45956f4822b1c5e81eefe212ae: stopping tablet replica
I20260620 20:15:48.522567 18906 raft_consensus.cc:2229] T 85250d317c554a5cb5a4942faa24290c P f6c53b45956f4822b1c5e81eefe212ae [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 20:15:48.523216 18906 raft_consensus.cc:2258] T 85250d317c554a5cb5a4942faa24290c P f6c53b45956f4822b1c5e81eefe212ae [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:15:48.524430 18512 tablet_service.cc:1467] Processing CreateTablet for tablet 5da92edae7ac4f4d9630d6ed06dc84dd (DEFAULT_TABLE table=test-table [id=799a266d254d4c8c909681eeddaed86c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:15:48.525079 18512 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5da92edae7ac4f4d9630d6ed06dc84dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:15:48.525182 18651 tablet_service.cc:1467] Processing CreateTablet for tablet 5da92edae7ac4f4d9630d6ed06dc84dd (DEFAULT_TABLE table=test-table [id=799a266d254d4c8c909681eeddaed86c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:15:48.526082 18651 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5da92edae7ac4f4d9630d6ed06dc84dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:15:48.532620 18882 tablet_bootstrap.cc:492] T 5da92edae7ac4f4d9630d6ed06dc84dd P eaf918de62a94ff88d3d82f3fbaf1b61: Bootstrap starting.
I20260620 20:15:48.535259 18906 ts_tablet_manager.cc:1905] T 85250d317c554a5cb5a4942faa24290c P f6c53b45956f4822b1c5e81eefe212ae: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:15:48.535341 18882 tablet_bootstrap.cc:654] T 5da92edae7ac4f4d9630d6ed06dc84dd P eaf918de62a94ff88d3d82f3fbaf1b61: Neither blocks nor log segments found. Creating new log.
I20260620 20:15:48.538499 18791 tablet_service.cc:1467] Processing CreateTablet for tablet 5da92edae7ac4f4d9630d6ed06dc84dd (DEFAULT_TABLE table=test-table [id=799a266d254d4c8c909681eeddaed86c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:15:48.538997 18791 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5da92edae7ac4f4d9630d6ed06dc84dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:15:48.540397 18880 tablet_bootstrap.cc:492] T 5da92edae7ac4f4d9630d6ed06dc84dd P 03c1ea73e1504bf9b87e1ac2bb358219: Bootstrap starting.
I20260620 20:15:48.542246 18880 tablet_bootstrap.cc:654] T 5da92edae7ac4f4d9630d6ed06dc84dd P 03c1ea73e1504bf9b87e1ac2bb358219: Neither blocks nor log segments found. Creating new log.
I20260620 20:15:48.544776 18906 ts_tablet_manager.cc:1918] T 85250d317c554a5cb5a4942faa24290c P f6c53b45956f4822b1c5e81eefe212ae: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 20:15:48.545068 18906 log.cc:1196] T 85250d317c554a5cb5a4942faa24290c P f6c53b45956f4822b1c5e81eefe212ae: Deleting WAL directory at /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986536687152-17252-0/minicluster-data/ts-2/wal/wals/85250d317c554a5cb5a4942faa24290c
I20260620 20:15:48.547145 18882 tablet_bootstrap.cc:492] T 5da92edae7ac4f4d9630d6ed06dc84dd P eaf918de62a94ff88d3d82f3fbaf1b61: No bootstrap required, opened a new log
I20260620 20:15:48.547403 18882 ts_tablet_manager.cc:1397] T 5da92edae7ac4f4d9630d6ed06dc84dd P eaf918de62a94ff88d3d82f3fbaf1b61: Time spent bootstrapping tablet: real 0.015s user 0.000s sys 0.008s
I20260620 20:15:48.548867 18882 raft_consensus.cc:348] T 5da92edae7ac4f4d9630d6ed06dc84dd P eaf918de62a94ff88d3d82f3fbaf1b61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaf918de62a94ff88d3d82f3fbaf1b61" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 43825 } } peers { permanent_uuid: "03c1ea73e1504bf9b87e1ac2bb358219" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42015 } } peers { permanent_uuid: "f6c53b45956f4822b1c5e81eefe212ae" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42419 } }
I20260620 20:15:48.549201 18882 raft_consensus.cc:374] T 5da92edae7ac4f4d9630d6ed06dc84dd P eaf918de62a94ff88d3d82f3fbaf1b61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:15:48.549387 18882 raft_consensus.cc:729] T 5da92edae7ac4f4d9630d6ed06dc84dd P eaf918de62a94ff88d3d82f3fbaf1b61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eaf918de62a94ff88d3d82f3fbaf1b61, State: Initialized, Role: FOLLOWER
I20260620 20:15:48.549821 18881 tablet_bootstrap.cc:492] T 5da92edae7ac4f4d9630d6ed06dc84dd P f6c53b45956f4822b1c5e81eefe212ae: Bootstrap starting.
I20260620 20:15:48.549780 18882 consensus_queue.cc:260] T 5da92edae7ac4f4d9630d6ed06dc84dd P eaf918de62a94ff88d3d82f3fbaf1b61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaf918de62a94ff88d3d82f3fbaf1b61" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 43825 } } peers { permanent_uuid: "03c1ea73e1504bf9b87e1ac2bb358219" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42015 } } peers { permanent_uuid: "f6c53b45956f4822b1c5e81eefe212ae" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42419 } }
I20260620 20:15:48.545903 18906 ts_tablet_manager.cc:1939] T 85250d317c554a5cb5a4942faa24290c P f6c53b45956f4822b1c5e81eefe212ae: Deleting consensus metadata
I20260620 20:15:48.551163 18905 tablet_replica.cc:330] T 85250d317c554a5cb5a4942faa24290c P eaf918de62a94ff88d3d82f3fbaf1b61: stopping tablet replica
I20260620 20:15:48.551523 18881 tablet_bootstrap.cc:654] T 5da92edae7ac4f4d9630d6ed06dc84dd P f6c53b45956f4822b1c5e81eefe212ae: Neither blocks nor log segments found. Creating new log.
I20260620 20:15:48.551635 18905 raft_consensus.cc:2229] T 85250d317c554a5cb5a4942faa24290c P eaf918de62a94ff88d3d82f3fbaf1b61 [term 1 LEADER]: Raft consensus shutting down.
I20260620 20:15:48.552685 18904 tablet_replica.cc:330] T 85250d317c554a5cb5a4942faa24290c P 03c1ea73e1504bf9b87e1ac2bb358219: stopping tablet replica
I20260620 20:15:48.552927 18359 catalog_manager.cc:4841] TS f6c53b45956f4822b1c5e81eefe212ae (127.16.217.3:42419): tablet 85250d317c554a5cb5a4942faa24290c (table test-table [id=679ba587a14e469c9ce91fd6a705d0f4]) successfully deleted
I20260620 20:15:48.553150 18904 raft_consensus.cc:2229] T 85250d317c554a5cb5a4942faa24290c P 03c1ea73e1504bf9b87e1ac2bb358219 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 20:15:48.553407 18882 ts_tablet_manager.cc:1428] T 5da92edae7ac4f4d9630d6ed06dc84dd P eaf918de62a94ff88d3d82f3fbaf1b61: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20260620 20:15:48.553731 18904 raft_consensus.cc:2258] T 85250d317c554a5cb5a4942faa24290c P 03c1ea73e1504bf9b87e1ac2bb358219 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:15:48.554414 18905 raft_consensus.cc:2258] T 85250d317c554a5cb5a4942faa24290c P eaf918de62a94ff88d3d82f3fbaf1b61 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:15:48.556197 18904 ts_tablet_manager.cc:1905] T 85250d317c554a5cb5a4942faa24290c P 03c1ea73e1504bf9b87e1ac2bb358219: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:15:48.556663 18880 tablet_bootstrap.cc:492] T 5da92edae7ac4f4d9630d6ed06dc84dd P 03c1ea73e1504bf9b87e1ac2bb358219: No bootstrap required, opened a new log
I20260620 20:15:48.556865 18880 ts_tablet_manager.cc:1397] T 5da92edae7ac4f4d9630d6ed06dc84dd P 03c1ea73e1504bf9b87e1ac2bb358219: Time spent bootstrapping tablet: real 0.017s user 0.005s sys 0.003s
I20260620 20:15:48.558252 18880 raft_consensus.cc:348] T 5da92edae7ac4f4d9630d6ed06dc84dd P 03c1ea73e1504bf9b87e1ac2bb358219 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaf918de62a94ff88d3d82f3fbaf1b61" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 43825 } } peers { permanent_uuid: "03c1ea73e1504bf9b87e1ac2bb358219" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42015 } } peers { permanent_uuid: "f6c53b45956f4822b1c5e81eefe212ae" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42419 } }
I20260620 20:15:48.559130 18881 tablet_bootstrap.cc:492] T 5da92edae7ac4f4d9630d6ed06dc84dd P f6c53b45956f4822b1c5e81eefe212ae: No bootstrap required, opened a new log
I20260620 20:15:48.559355 18881 ts_tablet_manager.cc:1397] T 5da92edae7ac4f4d9630d6ed06dc84dd P f6c53b45956f4822b1c5e81eefe212ae: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.001s
I20260620 20:15:48.560410 18905 ts_tablet_manager.cc:1905] T 85250d317c554a5cb5a4942faa24290c P eaf918de62a94ff88d3d82f3fbaf1b61: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:15:48.560649 18880 raft_consensus.cc:374] T 5da92edae7ac4f4d9630d6ed06dc84dd P 03c1ea73e1504bf9b87e1ac2bb358219 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:15:48.560690 18881 raft_consensus.cc:348] T 5da92edae7ac4f4d9630d6ed06dc84dd P f6c53b45956f4822b1c5e81eefe212ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaf918de62a94ff88d3d82f3fbaf1b61" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 43825 } } peers { permanent_uuid: "03c1ea73e1504bf9b87e1ac2bb358219" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42015 } } peers { permanent_uuid: "f6c53b45956f4822b1c5e81eefe212ae" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42419 } }
I20260620 20:15:48.560842 18880 raft_consensus.cc:729] T 5da92edae7ac4f4d9630d6ed06dc84dd P 03c1ea73e1504bf9b87e1ac2bb358219 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03c1ea73e1504bf9b87e1ac2bb358219, State: Initialized, Role: FOLLOWER
I20260620 20:15:48.561260 18880 consensus_queue.cc:260] T 5da92edae7ac4f4d9630d6ed06dc84dd P 03c1ea73e1504bf9b87e1ac2bb358219 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaf918de62a94ff88d3d82f3fbaf1b61" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 43825 } } peers { permanent_uuid: "03c1ea73e1504bf9b87e1ac2bb358219" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42015 } } peers { permanent_uuid: "f6c53b45956f4822b1c5e81eefe212ae" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42419 } }
I20260620 20:15:48.561637 18881 raft_consensus.cc:374] T 5da92edae7ac4f4d9630d6ed06dc84dd P f6c53b45956f4822b1c5e81eefe212ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:15:48.561785 18881 raft_consensus.cc:729] T 5da92edae7ac4f4d9630d6ed06dc84dd P f6c53b45956f4822b1c5e81eefe212ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6c53b45956f4822b1c5e81eefe212ae, State: Initialized, Role: FOLLOWER
I20260620 20:15:48.562129 18881 consensus_queue.cc:260] T 5da92edae7ac4f4d9630d6ed06dc84dd P f6c53b45956f4822b1c5e81eefe212ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaf918de62a94ff88d3d82f3fbaf1b61" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 43825 } } peers { permanent_uuid: "03c1ea73e1504bf9b87e1ac2bb358219" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42015 } } peers { permanent_uuid: "f6c53b45956f4822b1c5e81eefe212ae" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42419 } }
I20260620 20:15:48.564335 18880 ts_tablet_manager.cc:1428] T 5da92edae7ac4f4d9630d6ed06dc84dd P 03c1ea73e1504bf9b87e1ac2bb358219: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20260620 20:15:48.564384 18904 ts_tablet_manager.cc:1918] T 85250d317c554a5cb5a4942faa24290c P 03c1ea73e1504bf9b87e1ac2bb358219: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 20:15:48.564607 18904 log.cc:1196] T 85250d317c554a5cb5a4942faa24290c P 03c1ea73e1504bf9b87e1ac2bb358219: Deleting WAL directory at /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986536687152-17252-0/minicluster-data/ts-1/wal/wals/85250d317c554a5cb5a4942faa24290c
I20260620 20:15:48.564766 18881 ts_tablet_manager.cc:1428] T 5da92edae7ac4f4d9630d6ed06dc84dd P f6c53b45956f4822b1c5e81eefe212ae: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20260620 20:15:48.565418 18904 ts_tablet_manager.cc:1939] T 85250d317c554a5cb5a4942faa24290c P 03c1ea73e1504bf9b87e1ac2bb358219: Deleting consensus metadata
I20260620 20:15:48.567435 18361 catalog_manager.cc:4841] TS 03c1ea73e1504bf9b87e1ac2bb358219 (127.16.217.2:42015): tablet 85250d317c554a5cb5a4942faa24290c (table test-table [id=679ba587a14e469c9ce91fd6a705d0f4]) successfully deleted
I20260620 20:15:48.574532 18905 ts_tablet_manager.cc:1918] T 85250d317c554a5cb5a4942faa24290c P eaf918de62a94ff88d3d82f3fbaf1b61: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 20:15:48.574707 18905 log.cc:1196] T 85250d317c554a5cb5a4942faa24290c P eaf918de62a94ff88d3d82f3fbaf1b61: Deleting WAL directory at /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986536687152-17252-0/minicluster-data/ts-0/wal/wals/85250d317c554a5cb5a4942faa24290c
I20260620 20:15:48.575256 18905 ts_tablet_manager.cc:1939] T 85250d317c554a5cb5a4942faa24290c P eaf918de62a94ff88d3d82f3fbaf1b61: Deleting consensus metadata
I20260620 20:15:48.576956 18359 catalog_manager.cc:4841] TS eaf918de62a94ff88d3d82f3fbaf1b61 (127.16.217.1:43825): tablet 85250d317c554a5cb5a4942faa24290c (table test-table [id=679ba587a14e469c9ce91fd6a705d0f4]) successfully deleted
I20260620 20:15:48.738790 18887 raft_consensus.cc:482] T 5da92edae7ac4f4d9630d6ed06dc84dd P f6c53b45956f4822b1c5e81eefe212ae [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 20:15:48.739120 18887 raft_consensus.cc:504] T 5da92edae7ac4f4d9630d6ed06dc84dd P f6c53b45956f4822b1c5e81eefe212ae [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaf918de62a94ff88d3d82f3fbaf1b61" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 43825 } } peers { permanent_uuid: "03c1ea73e1504bf9b87e1ac2bb358219" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42015 } } peers { permanent_uuid: "f6c53b45956f4822b1c5e81eefe212ae" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42419 } }
I20260620 20:15:48.740793 18887 leader_election.cc:288] T 5da92edae7ac4f4d9630d6ed06dc84dd P f6c53b45956f4822b1c5e81eefe212ae [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eaf918de62a94ff88d3d82f3fbaf1b61 (127.16.217.1:43825), 03c1ea73e1504bf9b87e1ac2bb358219 (127.16.217.2:42015)
I20260620 20:15:48.748889 18876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:48.741440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.217.3:44333 (local address 127.16.217.2:42015)
0620 20:15:48.741626 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0620 20:15:48.741632 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:15:48.742672 (+ 1040us) server_negotiation.cc:407] Connection header received
0620 20:15:48.742900 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:15:48.742906 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:15:48.742983 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:15:48.743065 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:15:48.743768 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:48.744871 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:48.747100 (+ 2229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:48.747857 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:48.748024 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:15:48.748097 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 20:15:48.748436 (+ 339us) server_negotiation.cc:299] Negotiation successful
0620 20:15:48.748575 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 20:15:48.749146 18911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:48.741037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.1:43825 (local address 127.16.217.3:50701)
0620 20:15:48.742413 (+ 1376us) negotiation.cc:107] Waiting for socket to connect
0620 20:15:48.742433 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 20:15:48.742617 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:15:48.743057 (+ 440us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:15:48.743062 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:15:48.743089 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:15:48.743358 (+ 269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:15:48.743374 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:15:48.744592 (+ 1218us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:15:48.744599 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:15:48.746679 (+ 2080us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:15:48.746692 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:15:48.748366 (+ 1674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:15:48.748375 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:15:48.748548 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:15:48.748577 (+ 29us) client_negotiation.cc:769] Sending connection context
0620 20:15:48.748663 (+ 86us) client_negotiation.cc:240] Negotiation successful
0620 20:15:48.748793 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1250,"mutex_wait_us":48,"thread_start_us":1270,"threads_started":1}
I20260620 20:15:48.749157 18879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:48.741275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.217.3:50701 (local address 127.16.217.1:43825)
0620 20:15:48.741523 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0620 20:15:48.741532 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 20:15:48.742543 (+ 1011us) server_negotiation.cc:407] Connection header received
0620 20:15:48.742815 (+ 272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:15:48.742819 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:15:48.742918 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:15:48.743014 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:15:48.743530 (+ 516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:48.744450 (+ 920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:48.746843 (+ 2393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:48.748135 (+ 1292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:48.748301 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:15:48.748388 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0620 20:15:48.748702 (+ 314us) server_negotiation.cc:299] Negotiation successful
0620 20:15:48.748836 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260620 20:15:48.749934 18910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:48.741064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.2:42015 (local address 127.16.217.3:44333)
0620 20:15:48.742552 (+ 1488us) negotiation.cc:107] Waiting for socket to connect
0620 20:15:48.742571 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 20:15:48.742702 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:15:48.743221 (+ 519us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:15:48.743225 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:15:48.743245 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:15:48.743591 (+ 346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:15:48.743604 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:15:48.745006 (+ 1402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:15:48.745012 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:15:48.746962 (+ 1950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:15:48.746971 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:15:48.748025 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:15:48.748031 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:15:48.748193 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:15:48.748247 (+ 54us) client_negotiation.cc:769] Sending connection context
0620 20:15:48.749431 (+ 1184us) client_negotiation.cc:240] Negotiation successful
0620 20:15:48.749575 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1387,"mutex_wait_us":25,"thread_start_us":602,"threads_started":1}
I20260620 20:15:48.750571 18533 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5da92edae7ac4f4d9630d6ed06dc84dd" candidate_uuid: "f6c53b45956f4822b1c5e81eefe212ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaf918de62a94ff88d3d82f3fbaf1b61" is_pre_election: true
I20260620 20:15:48.750988 18677 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5da92edae7ac4f4d9630d6ed06dc84dd" candidate_uuid: "f6c53b45956f4822b1c5e81eefe212ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "03c1ea73e1504bf9b87e1ac2bb358219" is_pre_election: true
I20260620 20:15:48.751139 18533 raft_consensus.cc:2454] T 5da92edae7ac4f4d9630d6ed06dc84dd P eaf918de62a94ff88d3d82f3fbaf1b61 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f6c53b45956f4822b1c5e81eefe212ae in term 0.
I20260620 20:15:48.751233 18677 raft_consensus.cc:2454] T 5da92edae7ac4f4d9630d6ed06dc84dd P 03c1ea73e1504bf9b87e1ac2bb358219 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f6c53b45956f4822b1c5e81eefe212ae in term 0.
I20260620 20:15:48.751713 18744 leader_election.cc:302] T 5da92edae7ac4f4d9630d6ed06dc84dd P f6c53b45956f4822b1c5e81eefe212ae [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 03c1ea73e1504bf9b87e1ac2bb358219, f6c53b45956f4822b1c5e81eefe212ae; no voters:
I20260620 20:15:48.752126 18887 raft_consensus.cc:2781] T 5da92edae7ac4f4d9630d6ed06dc84dd P f6c53b45956f4822b1c5e81eefe212ae [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 20:15:48.752270 18887 raft_consensus.cc:482] T 5da92edae7ac4f4d9630d6ed06dc84dd P f6c53b45956f4822b1c5e81eefe212ae [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 20:15:48.752394 18887 raft_consensus.cc:3037] T 5da92edae7ac4f4d9630d6ed06dc84dd P f6c53b45956f4822b1c5e81eefe212ae [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:15:48.754606 18887 raft_consensus.cc:504] T 5da92edae7ac4f4d9630d6ed06dc84dd P f6c53b45956f4822b1c5e81eefe212ae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaf918de62a94ff88d3d82f3fbaf1b61" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 43825 } } peers { permanent_uuid: "03c1ea73e1504bf9b87e1ac2bb358219" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42015 } } peers { permanent_uuid: "f6c53b45956f4822b1c5e81eefe212ae" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42419 } }
I20260620 20:15:48.755282 18887 leader_election.cc:288] T 5da92edae7ac4f4d9630d6ed06dc84dd P f6c53b45956f4822b1c5e81eefe212ae [CANDIDATE]: Term 1 election: Requested vote from peers eaf918de62a94ff88d3d82f3fbaf1b61 (127.16.217.1:43825), 03c1ea73e1504bf9b87e1ac2bb358219 (127.16.217.2:42015)
I20260620 20:15:48.755735 18533 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5da92edae7ac4f4d9630d6ed06dc84dd" candidate_uuid: "f6c53b45956f4822b1c5e81eefe212ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaf918de62a94ff88d3d82f3fbaf1b61"
I20260620 20:15:48.755846 18677 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5da92edae7ac4f4d9630d6ed06dc84dd" candidate_uuid: "f6c53b45956f4822b1c5e81eefe212ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "03c1ea73e1504bf9b87e1ac2bb358219"
I20260620 20:15:48.755956 18533 raft_consensus.cc:3037] T 5da92edae7ac4f4d9630d6ed06dc84dd P eaf918de62a94ff88d3d82f3fbaf1b61 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:15:48.756052 18677 raft_consensus.cc:3037] T 5da92edae7ac4f4d9630d6ed06dc84dd P 03c1ea73e1504bf9b87e1ac2bb358219 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:15:48.759092 18677 raft_consensus.cc:2454] T 5da92edae7ac4f4d9630d6ed06dc84dd P 03c1ea73e1504bf9b87e1ac2bb358219 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6c53b45956f4822b1c5e81eefe212ae in term 1.
I20260620 20:15:48.759537 18744 leader_election.cc:302] T 5da92edae7ac4f4d9630d6ed06dc84dd P f6c53b45956f4822b1c5e81eefe212ae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 03c1ea73e1504bf9b87e1ac2bb358219, f6c53b45956f4822b1c5e81eefe212ae; no voters:
I20260620 20:15:48.759735 18533 raft_consensus.cc:2454] T 5da92edae7ac4f4d9630d6ed06dc84dd P eaf918de62a94ff88d3d82f3fbaf1b61 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6c53b45956f4822b1c5e81eefe212ae in term 1.
I20260620 20:15:48.759912 18887 raft_consensus.cc:2781] T 5da92edae7ac4f4d9630d6ed06dc84dd P f6c53b45956f4822b1c5e81eefe212ae [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:15:48.760365 18887 raft_consensus.cc:686] T 5da92edae7ac4f4d9630d6ed06dc84dd P f6c53b45956f4822b1c5e81eefe212ae [term 1 LEADER]: Becoming Leader. State: Replica: f6c53b45956f4822b1c5e81eefe212ae, State: Running, Role: LEADER
I20260620 20:15:48.760821 18887 consensus_queue.cc:237] T 5da92edae7ac4f4d9630d6ed06dc84dd P f6c53b45956f4822b1c5e81eefe212ae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaf918de62a94ff88d3d82f3fbaf1b61" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 43825 } } peers { permanent_uuid: "03c1ea73e1504bf9b87e1ac2bb358219" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42015 } } peers { permanent_uuid: "f6c53b45956f4822b1c5e81eefe212ae" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42419 } }
I20260620 20:15:48.765082 18375 catalog_manager.cc:5495] T 5da92edae7ac4f4d9630d6ed06dc84dd P f6c53b45956f4822b1c5e81eefe212ae reported cstate change: term changed from 0 to 1, leader changed from <none> to f6c53b45956f4822b1c5e81eefe212ae (127.16.217.3). New cstate: current_term: 1 leader_uuid: "f6c53b45956f4822b1c5e81eefe212ae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaf918de62a94ff88d3d82f3fbaf1b61" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 43825 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "03c1ea73e1504bf9b87e1ac2bb358219" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42015 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f6c53b45956f4822b1c5e81eefe212ae" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42419 } health_report { overall_health: HEALTHY } } }
Jun 20 20:15:48 dist-test-slave-2hpg krb5kdc[18330](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986546, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.217.62@KRBTEST.COM for kudu/127.16.217.62@KRBTEST.COM
I20260620 20:15:48.839154 18919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:48.824352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51456 (local address 127.16.217.62:34083)
0620 20:15:48.825135 (+ 783us) server_negotiation.cc:206] Beginning negotiation
0620 20:15:48.825141 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:15:48.825332 (+ 191us) server_negotiation.cc:407] Connection header received
0620 20:15:48.825536 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:15:48.825546 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:15:48.825596 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:15:48.825704 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:15:48.826759 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:48.827485 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:48.828691 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:48.829113 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:48.829241 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:15:48.833310 (+ 4069us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:15:48.833314 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:15:48.833317 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:15:48.833350 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:15:48.835962 (+ 2612us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:15:48.837104 (+ 1142us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:15:48.837111 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:15:48.837115 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:15:48.837237 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:15:48.837592 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:15:48.837600 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:15:48.837603 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:15:48.837846 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:15:48.838054 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0620 20:15:48.838451 (+ 397us) server_negotiation.cc:299] Negotiation successful
0620 20:15:48.838777 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":145,"server-negotiator.queue_time_us":611,"thread_start_us":377,"threads_started":1}
I20260620 20:15:48.839151 18920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:48.824367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:34083 (local address 127.0.0.1:51456)
0620 20:15:48.825226 (+ 859us) negotiation.cc:107] Waiting for socket to connect
0620 20:15:48.825241 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 20:15:48.825413 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:15:48.825732 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:15:48.825737 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:15:48.826439 (+ 702us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:15:48.826616 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:15:48.826625 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:15:48.827612 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:15:48.827621 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:15:48.828561 (+ 940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:15:48.828571 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:15:48.829286 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:15:48.829292 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:15:48.829968 (+ 676us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:15:48.830746 (+ 778us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:15:48.830772 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:15:48.833123 (+ 2351us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:15:48.836123 (+ 3000us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:15:48.836135 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:15:48.836143 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:15:48.836985 (+ 842us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:15:48.837339 (+ 354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:15:48.837346 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:15:48.837349 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:15:48.837490 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:15:48.838027 (+ 537us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:15:48.838035 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:15:48.838203 (+ 168us) client_negotiation.cc:769] Sending connection context
0620 20:15:48.838361 (+ 158us) client_negotiation.cc:240] Negotiation successful
0620 20:15:48.838658 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":743,"thread_start_us":609,"threads_started":1}
I20260620 20:15:48.845549 18375 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.16.217.62@KRBTEST.COM'} at 127.0.0.1:51456:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260620 20:15:48.846686 18375 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 20:15:48.854001 18513 tablet_service.cc:1467] Processing CreateTablet for tablet 8a983205b00e41a59fd599f119e3e890 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=8ab917349ae441c3aac9a3febb973087]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 20:15:48.854364 18791 tablet_service.cc:1467] Processing CreateTablet for tablet 8a983205b00e41a59fd599f119e3e890 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=8ab917349ae441c3aac9a3febb973087]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 20:15:48.854341 18652 tablet_service.cc:1467] Processing CreateTablet for tablet 8a983205b00e41a59fd599f119e3e890 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=8ab917349ae441c3aac9a3febb973087]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 20:15:48.854650 18513 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a983205b00e41a59fd599f119e3e890. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:15:48.854897 18791 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a983205b00e41a59fd599f119e3e890. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:15:48.854912 18652 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a983205b00e41a59fd599f119e3e890. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:15:48.862424 18881 tablet_bootstrap.cc:492] T 8a983205b00e41a59fd599f119e3e890 P f6c53b45956f4822b1c5e81eefe212ae: Bootstrap starting.
I20260620 20:15:48.862926 18880 tablet_bootstrap.cc:492] T 8a983205b00e41a59fd599f119e3e890 P 03c1ea73e1504bf9b87e1ac2bb358219: Bootstrap starting.
I20260620 20:15:48.864156 18881 tablet_bootstrap.cc:654] T 8a983205b00e41a59fd599f119e3e890 P f6c53b45956f4822b1c5e81eefe212ae: Neither blocks nor log segments found. Creating new log.
I20260620 20:15:48.864259 18880 tablet_bootstrap.cc:654] T 8a983205b00e41a59fd599f119e3e890 P 03c1ea73e1504bf9b87e1ac2bb358219: Neither blocks nor log segments found. Creating new log.
I20260620 20:15:48.867151 18880 tablet_bootstrap.cc:492] T 8a983205b00e41a59fd599f119e3e890 P 03c1ea73e1504bf9b87e1ac2bb358219: No bootstrap required, opened a new log
I20260620 20:15:48.867336 18880 ts_tablet_manager.cc:1397] T 8a983205b00e41a59fd599f119e3e890 P 03c1ea73e1504bf9b87e1ac2bb358219: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.001s
I20260620 20:15:48.867985 18880 raft_consensus.cc:348] T 8a983205b00e41a59fd599f119e3e890 P 03c1ea73e1504bf9b87e1ac2bb358219 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaf918de62a94ff88d3d82f3fbaf1b61" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 43825 } } peers { permanent_uuid: "03c1ea73e1504bf9b87e1ac2bb358219" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42015 } } peers { permanent_uuid: "f6c53b45956f4822b1c5e81eefe212ae" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42419 } }
I20260620 20:15:48.868340 18880 raft_consensus.cc:374] T 8a983205b00e41a59fd599f119e3e890 P 03c1ea73e1504bf9b87e1ac2bb358219 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:15:48.868419 18880 raft_consensus.cc:729] T 8a983205b00e41a59fd599f119e3e890 P 03c1ea73e1504bf9b87e1ac2bb358219 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03c1ea73e1504bf9b87e1ac2bb358219, State: Initialized, Role: FOLLOWER
I20260620 20:15:48.868551 18882 tablet_bootstrap.cc:492] T 8a983205b00e41a59fd599f119e3e890 P eaf918de62a94ff88d3d82f3fbaf1b61: Bootstrap starting.
I20260620 20:15:48.868728 18880 consensus_queue.cc:260] T 8a983205b00e41a59fd599f119e3e890 P 03c1ea73e1504bf9b87e1ac2bb358219 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaf918de62a94ff88d3d82f3fbaf1b61" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 43825 } } peers { permanent_uuid: "03c1ea73e1504bf9b87e1ac2bb358219" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42015 } } peers { permanent_uuid: "f6c53b45956f4822b1c5e81eefe212ae" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42419 } }
I20260620 20:15:48.869208 18886 tablet_replica.cc:439] T 8a983205b00e41a59fd599f119e3e890 P 03c1ea73e1504bf9b87e1ac2bb358219: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaf918de62a94ff88d3d82f3fbaf1b61" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 43825 } } peers { permanent_uuid: "03c1ea73e1504bf9b87e1ac2bb358219" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42015 } } peers { permanent_uuid: "f6c53b45956f4822b1c5e81eefe212ae" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42419 } } }
I20260620 20:15:48.869604 18886 tablet_replica.cc:442] T 8a983205b00e41a59fd599f119e3e890 P 03c1ea73e1504bf9b87e1ac2bb358219: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 20:15:48.870805 18882 tablet_bootstrap.cc:654] T 8a983205b00e41a59fd599f119e3e890 P eaf918de62a94ff88d3d82f3fbaf1b61: Neither blocks nor log segments found. Creating new log.
I20260620 20:15:48.873044 18880 ts_tablet_manager.cc:1428] T 8a983205b00e41a59fd599f119e3e890 P 03c1ea73e1504bf9b87e1ac2bb358219: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20260620 20:15:48.875808 18881 tablet_bootstrap.cc:492] T 8a983205b00e41a59fd599f119e3e890 P f6c53b45956f4822b1c5e81eefe212ae: No bootstrap required, opened a new log
I20260620 20:15:48.876031 18881 ts_tablet_manager.cc:1397] T 8a983205b00e41a59fd599f119e3e890 P f6c53b45956f4822b1c5e81eefe212ae: Time spent bootstrapping tablet: real 0.014s user 0.005s sys 0.006s
I20260620 20:15:48.876852 18881 raft_consensus.cc:348] T 8a983205b00e41a59fd599f119e3e890 P f6c53b45956f4822b1c5e81eefe212ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaf918de62a94ff88d3d82f3fbaf1b61" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 43825 } } peers { permanent_uuid: "03c1ea73e1504bf9b87e1ac2bb358219" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42015 } } peers { permanent_uuid: "f6c53b45956f4822b1c5e81eefe212ae" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42419 } }
I20260620 20:15:48.877254 18881 raft_consensus.cc:374] T 8a983205b00e41a59fd599f119e3e890 P f6c53b45956f4822b1c5e81eefe212ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:15:48.877398 18881 raft_consensus.cc:729] T 8a983205b00e41a59fd599f119e3e890 P f6c53b45956f4822b1c5e81eefe212ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6c53b45956f4822b1c5e81eefe212ae, State: Initialized, Role: FOLLOWER
I20260620 20:15:48.877694 18881 consensus_queue.cc:260] T 8a983205b00e41a59fd599f119e3e890 P f6c53b45956f4822b1c5e81eefe212ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaf918de62a94ff88d3d82f3fbaf1b61" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 43825 } } peers { permanent_uuid: "03c1ea73e1504bf9b87e1ac2bb358219" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42015 } } peers { permanent_uuid: "f6c53b45956f4822b1c5e81eefe212ae" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42419 } }
I20260620 20:15:48.878141 18887 tablet_replica.cc:439] T 8a983205b00e41a59fd599f119e3e890 P f6c53b45956f4822b1c5e81eefe212ae: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaf918de62a94ff88d3d82f3fbaf1b61" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 43825 } } peers { permanent_uuid: "03c1ea73e1504bf9b87e1ac2bb358219" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42015 } } peers { permanent_uuid: "f6c53b45956f4822b1c5e81eefe212ae" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42419 } } }
I20260620 20:15:48.878443 18887 tablet_replica.cc:442] T 8a983205b00e41a59fd599f119e3e890 P f6c53b45956f4822b1c5e81eefe212ae: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 20:15:48.878870 18882 tablet_bootstrap.cc:492] T 8a983205b00e41a59fd599f119e3e890 P eaf918de62a94ff88d3d82f3fbaf1b61: No bootstrap required, opened a new log
I20260620 20:15:48.879114 18882 ts_tablet_manager.cc:1397] T 8a983205b00e41a59fd599f119e3e890 P eaf918de62a94ff88d3d82f3fbaf1b61: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.004s
I20260620 20:15:48.879172 18881 ts_tablet_manager.cc:1428] T 8a983205b00e41a59fd599f119e3e890 P f6c53b45956f4822b1c5e81eefe212ae: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260620 20:15:48.880055 18882 raft_consensus.cc:348] T 8a983205b00e41a59fd599f119e3e890 P eaf918de62a94ff88d3d82f3fbaf1b61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaf918de62a94ff88d3d82f3fbaf1b61" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 43825 } } peers { permanent_uuid: "03c1ea73e1504bf9b87e1ac2bb358219" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42015 } } peers { permanent_uuid: "f6c53b45956f4822b1c5e81eefe212ae" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42419 } }
I20260620 20:15:48.880373 18882 raft_consensus.cc:374] T 8a983205b00e41a59fd599f119e3e890 P eaf918de62a94ff88d3d82f3fbaf1b61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:15:48.880506 18882 raft_consensus.cc:729] T 8a983205b00e41a59fd599f119e3e890 P eaf918de62a94ff88d3d82f3fbaf1b61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eaf918de62a94ff88d3d82f3fbaf1b61, State: Initialized, Role: FOLLOWER
I20260620 20:15:48.880820 18882 consensus_queue.cc:260] T 8a983205b00e41a59fd599f119e3e890 P eaf918de62a94ff88d3d82f3fbaf1b61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaf918de62a94ff88d3d82f3fbaf1b61" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 43825 } } peers { permanent_uuid: "03c1ea73e1504bf9b87e1ac2bb358219" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42015 } } peers { permanent_uuid: "f6c53b45956f4822b1c5e81eefe212ae" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42419 } }
I20260620 20:15:48.881371 18888 tablet_replica.cc:439] T 8a983205b00e41a59fd599f119e3e890 P eaf918de62a94ff88d3d82f3fbaf1b61: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaf918de62a94ff88d3d82f3fbaf1b61" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 43825 } } peers { permanent_uuid: "03c1ea73e1504bf9b87e1ac2bb358219" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42015 } } peers { permanent_uuid: "f6c53b45956f4822b1c5e81eefe212ae" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42419 } } }
I20260620 20:15:48.881681 18888 tablet_replica.cc:442] T 8a983205b00e41a59fd599f119e3e890 P eaf918de62a94ff88d3d82f3fbaf1b61: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 20:15:48.882774 18882 ts_tablet_manager.cc:1428] T 8a983205b00e41a59fd599f119e3e890 P eaf918de62a94ff88d3d82f3fbaf1b61: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20260620 20:15:49.039449 18887 raft_consensus.cc:482] T 8a983205b00e41a59fd599f119e3e890 P f6c53b45956f4822b1c5e81eefe212ae [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 20:15:49.039654 18887 raft_consensus.cc:504] T 8a983205b00e41a59fd599f119e3e890 P f6c53b45956f4822b1c5e81eefe212ae [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaf918de62a94ff88d3d82f3fbaf1b61" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 43825 } } peers { permanent_uuid: "03c1ea73e1504bf9b87e1ac2bb358219" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42015 } } peers { permanent_uuid: "f6c53b45956f4822b1c5e81eefe212ae" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42419 } }
I20260620 20:15:49.040350 18887 leader_election.cc:288] T 8a983205b00e41a59fd599f119e3e890 P f6c53b45956f4822b1c5e81eefe212ae [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eaf918de62a94ff88d3d82f3fbaf1b61 (127.16.217.1:43825), 03c1ea73e1504bf9b87e1ac2bb358219 (127.16.217.2:42015)
I20260620 20:15:49.041018 18677 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a983205b00e41a59fd599f119e3e890" candidate_uuid: "f6c53b45956f4822b1c5e81eefe212ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "03c1ea73e1504bf9b87e1ac2bb358219" is_pre_election: true
I20260620 20:15:49.041016 18533 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a983205b00e41a59fd599f119e3e890" candidate_uuid: "f6c53b45956f4822b1c5e81eefe212ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaf918de62a94ff88d3d82f3fbaf1b61" is_pre_election: true
I20260620 20:15:49.041276 18533 raft_consensus.cc:2454] T 8a983205b00e41a59fd599f119e3e890 P eaf918de62a94ff88d3d82f3fbaf1b61 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f6c53b45956f4822b1c5e81eefe212ae in term 0.
I20260620 20:15:49.041293 18677 raft_consensus.cc:2454] T 8a983205b00e41a59fd599f119e3e890 P 03c1ea73e1504bf9b87e1ac2bb358219 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f6c53b45956f4822b1c5e81eefe212ae in term 0.
I20260620 20:15:49.041734 18744 leader_election.cc:302] T 8a983205b00e41a59fd599f119e3e890 P f6c53b45956f4822b1c5e81eefe212ae [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 03c1ea73e1504bf9b87e1ac2bb358219, eaf918de62a94ff88d3d82f3fbaf1b61, f6c53b45956f4822b1c5e81eefe212ae; no voters:
I20260620 20:15:49.042266 18887 raft_consensus.cc:2781] T 8a983205b00e41a59fd599f119e3e890 P f6c53b45956f4822b1c5e81eefe212ae [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 20:15:49.042361 18887 raft_consensus.cc:482] T 8a983205b00e41a59fd599f119e3e890 P f6c53b45956f4822b1c5e81eefe212ae [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 20:15:49.042433 18887 raft_consensus.cc:3037] T 8a983205b00e41a59fd599f119e3e890 P f6c53b45956f4822b1c5e81eefe212ae [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:15:49.044345 18887 raft_consensus.cc:504] T 8a983205b00e41a59fd599f119e3e890 P f6c53b45956f4822b1c5e81eefe212ae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaf918de62a94ff88d3d82f3fbaf1b61" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 43825 } } peers { permanent_uuid: "03c1ea73e1504bf9b87e1ac2bb358219" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42015 } } peers { permanent_uuid: "f6c53b45956f4822b1c5e81eefe212ae" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42419 } }
I20260620 20:15:49.044898 18887 leader_election.cc:288] T 8a983205b00e41a59fd599f119e3e890 P f6c53b45956f4822b1c5e81eefe212ae [CANDIDATE]: Term 1 election: Requested vote from peers eaf918de62a94ff88d3d82f3fbaf1b61 (127.16.217.1:43825), 03c1ea73e1504bf9b87e1ac2bb358219 (127.16.217.2:42015)
I20260620 20:15:49.045343 18533 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a983205b00e41a59fd599f119e3e890" candidate_uuid: "f6c53b45956f4822b1c5e81eefe212ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaf918de62a94ff88d3d82f3fbaf1b61"
I20260620 20:15:49.045351 18677 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a983205b00e41a59fd599f119e3e890" candidate_uuid: "f6c53b45956f4822b1c5e81eefe212ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "03c1ea73e1504bf9b87e1ac2bb358219"
I20260620 20:15:49.045568 18677 raft_consensus.cc:3037] T 8a983205b00e41a59fd599f119e3e890 P 03c1ea73e1504bf9b87e1ac2bb358219 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:15:49.045568 18533 raft_consensus.cc:3037] T 8a983205b00e41a59fd599f119e3e890 P eaf918de62a94ff88d3d82f3fbaf1b61 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:15:49.048396 18677 raft_consensus.cc:2454] T 8a983205b00e41a59fd599f119e3e890 P 03c1ea73e1504bf9b87e1ac2bb358219 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6c53b45956f4822b1c5e81eefe212ae in term 1.
I20260620 20:15:49.048396 18533 raft_consensus.cc:2454] T 8a983205b00e41a59fd599f119e3e890 P eaf918de62a94ff88d3d82f3fbaf1b61 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6c53b45956f4822b1c5e81eefe212ae in term 1.
I20260620 20:15:49.048784 18744 leader_election.cc:302] T 8a983205b00e41a59fd599f119e3e890 P f6c53b45956f4822b1c5e81eefe212ae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 03c1ea73e1504bf9b87e1ac2bb358219, f6c53b45956f4822b1c5e81eefe212ae; no voters:
I20260620 20:15:49.049073 18887 raft_consensus.cc:2781] T 8a983205b00e41a59fd599f119e3e890 P f6c53b45956f4822b1c5e81eefe212ae [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:15:49.049257 18887 raft_consensus.cc:686] T 8a983205b00e41a59fd599f119e3e890 P f6c53b45956f4822b1c5e81eefe212ae [term 1 LEADER]: Becoming Leader. State: Replica: f6c53b45956f4822b1c5e81eefe212ae, State: Running, Role: LEADER
I20260620 20:15:49.049510 18887 consensus_queue.cc:237] T 8a983205b00e41a59fd599f119e3e890 P f6c53b45956f4822b1c5e81eefe212ae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaf918de62a94ff88d3d82f3fbaf1b61" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 43825 } } peers { permanent_uuid: "03c1ea73e1504bf9b87e1ac2bb358219" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42015 } } peers { permanent_uuid: "f6c53b45956f4822b1c5e81eefe212ae" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42419 } }
I20260620 20:15:49.051095 18912 tablet_replica.cc:439] T 8a983205b00e41a59fd599f119e3e890 P f6c53b45956f4822b1c5e81eefe212ae: TxnStatusTablet state changed. Reason: New leader f6c53b45956f4822b1c5e81eefe212ae. Latest consensus state: current_term: 1 leader_uuid: "f6c53b45956f4822b1c5e81eefe212ae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaf918de62a94ff88d3d82f3fbaf1b61" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 43825 } } peers { permanent_uuid: "03c1ea73e1504bf9b87e1ac2bb358219" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42015 } } peers { permanent_uuid: "f6c53b45956f4822b1c5e81eefe212ae" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42419 } } }
I20260620 20:15:49.051401 18912 tablet_replica.cc:442] T 8a983205b00e41a59fd599f119e3e890 P f6c53b45956f4822b1c5e81eefe212ae: This TxnStatusTablet replica's current role is: LEADER
I20260620 20:15:49.052045 18925 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260620 20:15:49.052273 18925 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260620 20:15:49.052699 18374 catalog_manager.cc:5495] T 8a983205b00e41a59fd599f119e3e890 P f6c53b45956f4822b1c5e81eefe212ae reported cstate change: term changed from 0 to 1, leader changed from <none> to f6c53b45956f4822b1c5e81eefe212ae (127.16.217.3). New cstate: current_term: 1 leader_uuid: "f6c53b45956f4822b1c5e81eefe212ae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaf918de62a94ff88d3d82f3fbaf1b61" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 43825 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "03c1ea73e1504bf9b87e1ac2bb358219" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42015 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f6c53b45956f4822b1c5e81eefe212ae" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42419 } health_report { overall_health: HEALTHY } } }
I20260620 20:15:49.085094 18920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:49.077590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.3:42419 (local address 127.0.0.1:60242)
0620 20:15:49.077987 (+ 397us) negotiation.cc:107] Waiting for socket to connect
0620 20:15:49.078016 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0620 20:15:49.078172 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:15:49.078747 (+ 575us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:15:49.078751 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:15:49.078761 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 20:15:49.078953 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:15:49.078962 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:15:49.080959 (+ 1997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:15:49.080963 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:15:49.082166 (+ 1203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:15:49.082179 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:15:49.083190 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:15:49.083195 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:15:49.083384 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:15:49.083444 (+ 60us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 20:15:49.084604 (+ 1160us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 20:15:49.084631 (+ 27us) client_negotiation.cc:769] Sending connection context
0620 20:15:49.084713 (+ 82us) client_negotiation.cc:240] Negotiation successful
0620 20:15:49.084861 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59}
I20260620 20:15:49.085229 18928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:49.077593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60242 (local address 127.16.217.3:42419)
0620 20:15:49.078281 (+ 688us) server_negotiation.cc:206] Beginning negotiation
0620 20:15:49.078287 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:15:49.078314 (+ 27us) server_negotiation.cc:407] Connection header received
0620 20:15:49.078423 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:15:49.078428 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:15:49.078541 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:15:49.078708 (+ 167us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 20:15:49.079125 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:49.080802 (+ 1677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:49.082307 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:49.083033 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:49.083165 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:15:49.083633 (+ 468us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 20:15:49.084469 (+ 836us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 20:15:49.084630 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0620 20:15:49.084773 (+ 143us) server_negotiation.cc:299] Negotiation successful
0620 20:15:49.084855 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":555,"thread_start_us":381,"threads_started":1}
I20260620 20:15:49.089608 18677 raft_consensus.cc:1261] T 8a983205b00e41a59fd599f119e3e890 P 03c1ea73e1504bf9b87e1ac2bb358219 [term 1 FOLLOWER]: Refusing update from remote peer f6c53b45956f4822b1c5e81eefe212ae: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:15:49.089704 18533 raft_consensus.cc:1261] T 8a983205b00e41a59fd599f119e3e890 P eaf918de62a94ff88d3d82f3fbaf1b61 [term 1 FOLLOWER]: Refusing update from remote peer f6c53b45956f4822b1c5e81eefe212ae: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:15:49.090474 18912 consensus_queue.cc:1035] T 8a983205b00e41a59fd599f119e3e890 P f6c53b45956f4822b1c5e81eefe212ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "03c1ea73e1504bf9b87e1ac2bb358219" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42015 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:15:49.090806 18887 consensus_queue.cc:1035] T 8a983205b00e41a59fd599f119e3e890 P f6c53b45956f4822b1c5e81eefe212ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "eaf918de62a94ff88d3d82f3fbaf1b61" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 43825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:15:49.108305 18886 tablet_replica.cc:439] T 8a983205b00e41a59fd599f119e3e890 P 03c1ea73e1504bf9b87e1ac2bb358219: TxnStatusTablet state changed. Reason: New leader f6c53b45956f4822b1c5e81eefe212ae. Latest consensus state: current_term: 1 leader_uuid: "f6c53b45956f4822b1c5e81eefe212ae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaf918de62a94ff88d3d82f3fbaf1b61" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 43825 } } peers { permanent_uuid: "03c1ea73e1504bf9b87e1ac2bb358219" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42015 } } peers { permanent_uuid: "f6c53b45956f4822b1c5e81eefe212ae" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42419 } } }
I20260620 20:15:49.108635 18886 tablet_replica.cc:442] T 8a983205b00e41a59fd599f119e3e890 P 03c1ea73e1504bf9b87e1ac2bb358219: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 20:15:49.123232 18888 tablet_replica.cc:439] T 8a983205b00e41a59fd599f119e3e890 P eaf918de62a94ff88d3d82f3fbaf1b61: TxnStatusTablet state changed. Reason: New leader f6c53b45956f4822b1c5e81eefe212ae. Latest consensus state: current_term: 1 leader_uuid: "f6c53b45956f4822b1c5e81eefe212ae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaf918de62a94ff88d3d82f3fbaf1b61" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 43825 } } peers { permanent_uuid: "03c1ea73e1504bf9b87e1ac2bb358219" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42015 } } peers { permanent_uuid: "f6c53b45956f4822b1c5e81eefe212ae" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42419 } } }
I20260620 20:15:49.123703 18888 tablet_replica.cc:442] T 8a983205b00e41a59fd599f119e3e890 P eaf918de62a94ff88d3d82f3fbaf1b61: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 20:15:49.137121 18887 tablet_replica.cc:439] T 8a983205b00e41a59fd599f119e3e890 P f6c53b45956f4822b1c5e81eefe212ae: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f6c53b45956f4822b1c5e81eefe212ae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaf918de62a94ff88d3d82f3fbaf1b61" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 43825 } } peers { permanent_uuid: "03c1ea73e1504bf9b87e1ac2bb358219" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42015 } } peers { permanent_uuid: "f6c53b45956f4822b1c5e81eefe212ae" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42419 } } }
I20260620 20:15:49.138365 18887 tablet_replica.cc:442] T 8a983205b00e41a59fd599f119e3e890 P f6c53b45956f4822b1c5e81eefe212ae: This TxnStatusTablet replica's current role is: LEADER
I20260620 20:15:49.138651 18887 tablet_replica.cc:439] T 8a983205b00e41a59fd599f119e3e890 P f6c53b45956f4822b1c5e81eefe212ae: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f6c53b45956f4822b1c5e81eefe212ae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaf918de62a94ff88d3d82f3fbaf1b61" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 43825 } } peers { permanent_uuid: "03c1ea73e1504bf9b87e1ac2bb358219" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42015 } } peers { permanent_uuid: "f6c53b45956f4822b1c5e81eefe212ae" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42419 } } }
I20260620 20:15:49.139236 18887 tablet_replica.cc:442] T 8a983205b00e41a59fd599f119e3e890 P f6c53b45956f4822b1c5e81eefe212ae: This TxnStatusTablet replica's current role is: LEADER
I20260620 20:15:49.147687 18886 tablet_replica.cc:439] T 8a983205b00e41a59fd599f119e3e890 P 03c1ea73e1504bf9b87e1ac2bb358219: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f6c53b45956f4822b1c5e81eefe212ae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaf918de62a94ff88d3d82f3fbaf1b61" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 43825 } } peers { permanent_uuid: "03c1ea73e1504bf9b87e1ac2bb358219" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42015 } } peers { permanent_uuid: "f6c53b45956f4822b1c5e81eefe212ae" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42419 } } }
I20260620 20:15:49.147859 18888 tablet_replica.cc:439] T 8a983205b00e41a59fd599f119e3e890 P eaf918de62a94ff88d3d82f3fbaf1b61: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f6c53b45956f4822b1c5e81eefe212ae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaf918de62a94ff88d3d82f3fbaf1b61" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 43825 } } peers { permanent_uuid: "03c1ea73e1504bf9b87e1ac2bb358219" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42015 } } peers { permanent_uuid: "f6c53b45956f4822b1c5e81eefe212ae" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42419 } } }
I20260620 20:15:49.148092 18888 tablet_replica.cc:442] T 8a983205b00e41a59fd599f119e3e890 P eaf918de62a94ff88d3d82f3fbaf1b61: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 20:15:49.148094 18886 tablet_replica.cc:442] T 8a983205b00e41a59fd599f119e3e890 P 03c1ea73e1504bf9b87e1ac2bb358219: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 20:15:49.171878 18928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:49.155264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60244 (local address 127.16.217.3:42419)
0620 20:15:49.155446 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0620 20:15:49.155453 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:15:49.164748 (+ 9295us) server_negotiation.cc:407] Connection header received
0620 20:15:49.165189 (+ 441us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:15:49.165196 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:15:49.165285 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:15:49.165368 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 20:15:49.166160 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:49.167418 (+ 1258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:49.169598 (+ 2180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:49.170321 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:49.170441 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:15:49.171038 (+ 597us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 20:15:49.171142 (+ 104us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 20:15:49.171266 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0620 20:15:49.171441 (+ 175us) server_negotiation.cc:299] Negotiation successful
0620 20:15:49.171523 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 20:15:49.191425 18928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:49.182946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.217.3:36871 (local address 127.16.217.3:42419)
0620 20:15:49.183160 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0620 20:15:49.183168 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 20:15:49.183920 (+ 752us) server_negotiation.cc:407] Connection header received
0620 20:15:49.184142 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:15:49.184148 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:15:49.184240 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:15:49.184377 (+ 137us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 20:15:49.184976 (+ 599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:49.185820 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:49.187977 (+ 2157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:49.188431 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:49.188582 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:15:49.190334 (+ 1752us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 20:15:49.190468 (+ 134us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 20:15:49.190569 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0620 20:15:49.190800 (+ 231us) server_negotiation.cc:299] Negotiation successful
0620 20:15:49.190915 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 20:15:49.191542 18942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:49.182989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.3:42419 (local address 127.16.217.3:36871)
0620 20:15:49.183811 (+ 822us) negotiation.cc:107] Waiting for socket to connect
0620 20:15:49.183835 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 20:15:49.184008 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:15:49.184421 (+ 413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:15:49.184428 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:15:49.184440 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 20:15:49.184808 (+ 368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:15:49.184820 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:15:49.186034 (+ 1214us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:15:49.186038 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:15:49.187819 (+ 1781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:15:49.187831 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:15:49.188629 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:15:49.188638 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:15:49.188843 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:15:49.189989 (+ 1146us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 20:15:49.190609 (+ 620us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 20:15:49.190624 (+ 15us) client_negotiation.cc:769] Sending connection context
0620 20:15:49.190753 (+ 129us) client_negotiation.cc:240] Negotiation successful
0620 20:15:49.190943 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":655,"thread_start_us":479,"threads_started":1}
I20260620 20:15:49.203014 18532 raft_consensus.cc:1261] T 5da92edae7ac4f4d9630d6ed06dc84dd P eaf918de62a94ff88d3d82f3fbaf1b61 [term 1 FOLLOWER]: Refusing update from remote peer f6c53b45956f4822b1c5e81eefe212ae: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:15:49.203034 18676 raft_consensus.cc:1261] T 5da92edae7ac4f4d9630d6ed06dc84dd P 03c1ea73e1504bf9b87e1ac2bb358219 [term 1 FOLLOWER]: Refusing update from remote peer f6c53b45956f4822b1c5e81eefe212ae: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:15:49.203655 18887 consensus_queue.cc:1035] T 5da92edae7ac4f4d9630d6ed06dc84dd P f6c53b45956f4822b1c5e81eefe212ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "03c1ea73e1504bf9b87e1ac2bb358219" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42015 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:15:49.204244 18912 consensus_queue.cc:1035] T 5da92edae7ac4f4d9630d6ed06dc84dd P f6c53b45956f4822b1c5e81eefe212ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "eaf918de62a94ff88d3d82f3fbaf1b61" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 43825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:15:49.264730 18375 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51448:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 20:15:49.264945 18375 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51448:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 20:15:49.267916 18375 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 95ec7c2914b94cc7a3ccadd2791d3836: Sending DeleteTablet for 3 replicas of tablet 5da92edae7ac4f4d9630d6ed06dc84dd
I20260620 20:15:49.268747 18513 tablet_service.cc:1514] Processing DeleteTablet for tablet 5da92edae7ac4f4d9630d6ed06dc84dd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:15:49 UTC) from {username='kudu', principal='kudu/127.16.217.62@KRBTEST.COM'} at 127.0.0.1:51432
I20260620 20:15:49.268834 18790 tablet_service.cc:1514] Processing DeleteTablet for tablet 5da92edae7ac4f4d9630d6ed06dc84dd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:15:49 UTC) from {username='kudu', principal='kudu/127.16.217.62@KRBTEST.COM'} at 127.0.0.1:60226
I20260620 20:15:49.268819 18652 tablet_service.cc:1514] Processing DeleteTablet for tablet 5da92edae7ac4f4d9630d6ed06dc84dd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:15:49 UTC) from {username='kudu', principal='kudu/127.16.217.62@KRBTEST.COM'} at 127.0.0.1:48548
I20260620 20:15:49.279529 18946 tablet_replica.cc:330] T 5da92edae7ac4f4d9630d6ed06dc84dd P eaf918de62a94ff88d3d82f3fbaf1b61: stopping tablet replica
I20260620 20:15:49.279873 18946 raft_consensus.cc:2229] T 5da92edae7ac4f4d9630d6ed06dc84dd P eaf918de62a94ff88d3d82f3fbaf1b61 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 20:15:49.280166 18946 raft_consensus.cc:2258] T 5da92edae7ac4f4d9630d6ed06dc84dd P eaf918de62a94ff88d3d82f3fbaf1b61 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:15:49.281203 18947 tablet_replica.cc:330] T 5da92edae7ac4f4d9630d6ed06dc84dd P 03c1ea73e1504bf9b87e1ac2bb358219: stopping tablet replica
I20260620 20:15:49.281447 18946 ts_tablet_manager.cc:1905] T 5da92edae7ac4f4d9630d6ed06dc84dd P eaf918de62a94ff88d3d82f3fbaf1b61: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:15:49.281466 18947 raft_consensus.cc:2229] T 5da92edae7ac4f4d9630d6ed06dc84dd P 03c1ea73e1504bf9b87e1ac2bb358219 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 20:15:49.281742 18947 raft_consensus.cc:2258] T 5da92edae7ac4f4d9630d6ed06dc84dd P 03c1ea73e1504bf9b87e1ac2bb358219 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:15:49.283185 18947 ts_tablet_manager.cc:1905] T 5da92edae7ac4f4d9630d6ed06dc84dd P 03c1ea73e1504bf9b87e1ac2bb358219: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:15:49.286104 18948 tablet_replica.cc:330] T 5da92edae7ac4f4d9630d6ed06dc84dd P f6c53b45956f4822b1c5e81eefe212ae: stopping tablet replica
I20260620 20:15:49.286365 18948 raft_consensus.cc:2229] T 5da92edae7ac4f4d9630d6ed06dc84dd P f6c53b45956f4822b1c5e81eefe212ae [term 1 LEADER]: Raft consensus shutting down.
I20260620 20:15:49.287025 18948 raft_consensus.cc:2258] T 5da92edae7ac4f4d9630d6ed06dc84dd P f6c53b45956f4822b1c5e81eefe212ae [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:15:49.288312 18948 ts_tablet_manager.cc:1905] T 5da92edae7ac4f4d9630d6ed06dc84dd P f6c53b45956f4822b1c5e81eefe212ae: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:15:49.290689 18947 ts_tablet_manager.cc:1918] T 5da92edae7ac4f4d9630d6ed06dc84dd P 03c1ea73e1504bf9b87e1ac2bb358219: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 20:15:49.290838 18947 log.cc:1196] T 5da92edae7ac4f4d9630d6ed06dc84dd P 03c1ea73e1504bf9b87e1ac2bb358219: Deleting WAL directory at /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986536687152-17252-0/minicluster-data/ts-1/wal/wals/5da92edae7ac4f4d9630d6ed06dc84dd
I20260620 20:15:49.291307 18947 ts_tablet_manager.cc:1939] T 5da92edae7ac4f4d9630d6ed06dc84dd P 03c1ea73e1504bf9b87e1ac2bb358219: Deleting consensus metadata
I20260620 20:15:49.292425 18361 catalog_manager.cc:4841] TS 03c1ea73e1504bf9b87e1ac2bb358219 (127.16.217.2:42015): tablet 5da92edae7ac4f4d9630d6ed06dc84dd (table test-table [id=799a266d254d4c8c909681eeddaed86c]) successfully deleted
I20260620 20:15:49.294636 18948 ts_tablet_manager.cc:1918] T 5da92edae7ac4f4d9630d6ed06dc84dd P f6c53b45956f4822b1c5e81eefe212ae: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 20:15:49.294783 18948 log.cc:1196] T 5da92edae7ac4f4d9630d6ed06dc84dd P f6c53b45956f4822b1c5e81eefe212ae: Deleting WAL directory at /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986536687152-17252-0/minicluster-data/ts-2/wal/wals/5da92edae7ac4f4d9630d6ed06dc84dd
I20260620 20:15:49.295228 18948 ts_tablet_manager.cc:1939] T 5da92edae7ac4f4d9630d6ed06dc84dd P f6c53b45956f4822b1c5e81eefe212ae: Deleting consensus metadata
I20260620 20:15:49.296193 18359 catalog_manager.cc:4841] TS f6c53b45956f4822b1c5e81eefe212ae (127.16.217.3:42419): tablet 5da92edae7ac4f4d9630d6ed06dc84dd (table test-table [id=799a266d254d4c8c909681eeddaed86c]) successfully deleted
I20260620 20:15:49.297642 18946 ts_tablet_manager.cc:1918] T 5da92edae7ac4f4d9630d6ed06dc84dd P eaf918de62a94ff88d3d82f3fbaf1b61: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 20:15:49.297780 18946 log.cc:1196] T 5da92edae7ac4f4d9630d6ed06dc84dd P eaf918de62a94ff88d3d82f3fbaf1b61: Deleting WAL directory at /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986536687152-17252-0/minicluster-data/ts-0/wal/wals/5da92edae7ac4f4d9630d6ed06dc84dd
I20260620 20:15:49.302505 18946 ts_tablet_manager.cc:1939] T 5da92edae7ac4f4d9630d6ed06dc84dd P eaf918de62a94ff88d3d82f3fbaf1b61: Deleting consensus metadata
I20260620 20:15:49.304113 18359 catalog_manager.cc:4841] TS eaf918de62a94ff88d3d82f3fbaf1b61 (127.16.217.1:43825): tablet 5da92edae7ac4f4d9630d6ed06dc84dd (table test-table [id=799a266d254d4c8c909681eeddaed86c]) successfully deleted
Jun 20 20:15:49 dist-test-slave-2hpg krb5kdc[18330](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986548, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.16.217.1@KRBTEST.COM
I20260620 20:15:49.319496 18950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:49.282239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51456 (local address 127.16.217.1:43825)
0620 20:15:49.301834 (+ 19595us) server_negotiation.cc:206] Beginning negotiation
0620 20:15:49.301843 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 20:15:49.301887 (+ 44us) server_negotiation.cc:407] Connection header received
0620 20:15:49.301955 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:15:49.301960 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:15:49.302046 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:15:49.302183 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:15:49.304696 (+ 2513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:49.305860 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:49.307215 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:49.307866 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:49.307995 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:15:49.312485 (+ 4490us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:15:49.312501 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:15:49.312517 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:15:49.312550 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:15:49.315920 (+ 3370us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:15:49.316878 (+ 958us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:15:49.316887 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:15:49.316892 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:15:49.317037 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:15:49.317528 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:15:49.317536 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:15:49.317542 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:15:49.318001 (+ 459us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:15:49.318226 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0620 20:15:49.319026 (+ 800us) server_negotiation.cc:299] Negotiation successful
0620 20:15:49.319202 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19447,"thread_start_us":20064,"threads_started":1}
W20260620 20:15:49.320300 18578 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51456
I20260620 20:15:49.337205 18919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:49.323182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51470 (local address 127.16.217.62:34083)
0620 20:15:49.323349 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0620 20:15:49.323353 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:15:49.323721 (+ 368us) server_negotiation.cc:407] Connection header received
0620 20:15:49.323949 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:15:49.323954 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:15:49.324016 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:15:49.324120 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:15:49.325279 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:49.326098 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:49.327337 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:49.327998 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:49.328121 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:15:49.331117 (+ 2996us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:15:49.331124 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:15:49.331129 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:15:49.331168 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:15:49.334507 (+ 3339us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:15:49.335208 (+ 701us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:15:49.335213 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:15:49.335217 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:15:49.335372 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:15:49.335727 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:15:49.335732 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:15:49.335736 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:15:49.336079 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:15:49.336319 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0620 20:15:49.336652 (+ 333us) server_negotiation.cc:299] Negotiation successful
0620 20:15:49.336870 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
W20260620 20:15:49.337754 18375 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51470
I20260620 20:15:49.340296 17252 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu pid 18447
I20260620 20:15:49.353351 18950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:49.341085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51468 (local address 127.16.217.1:43825)
0620 20:15:49.341433 (+ 348us) server_negotiation.cc:206] Beginning negotiation
0620 20:15:49.341439 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:15:49.341677 (+ 238us) server_negotiation.cc:407] Connection header received
0620 20:15:49.341911 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:15:49.341917 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:15:49.341989 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:15:49.342089 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:15:49.343163 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:49.343840 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:49.344975 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:49.345386 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:49.345515 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:15:49.347790 (+ 2275us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:15:49.347807 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:15:49.347820 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:15:49.347854 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:15:49.350649 (+ 2795us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:15:49.351316 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:15:49.351324 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:15:49.351335 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:15:49.351480 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:15:49.351883 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:15:49.351890 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:15:49.351894 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:15:49.352155 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:15:49.352422 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0620 20:15:49.352750 (+ 328us) server_negotiation.cc:299] Negotiation successful
0620 20:15:49.352985 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":200}
W20260620 20:15:49.353983 18578 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51468
W20260620 20:15:49.354530 17252 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu pid 18447: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 20:15:49.354691 17252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu with pid 18447
I20260620 20:15:49.383602 17252 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu pid 18587
Jun 20 20:15:49 dist-test-slave-2hpg krb5kdc[18330](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986548, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.16.217.2@KRBTEST.COM
I20260620 20:15:49.399662 18955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:49.384442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48564 (local address 127.16.217.2:42015)
0620 20:15:49.384899 (+ 457us) server_negotiation.cc:206] Beginning negotiation
0620 20:15:49.384904 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:15:49.384918 (+ 14us) server_negotiation.cc:407] Connection header received
0620 20:15:49.384969 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:15:49.384972 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:15:49.385029 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:15:49.385146 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:15:49.386366 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:49.387111 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:49.388423 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:49.388844 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:49.388970 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:15:49.393352 (+ 4382us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:15:49.393365 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:15:49.393378 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:15:49.393413 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:15:49.396591 (+ 3178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:15:49.397470 (+ 879us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:15:49.397477 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:15:49.397480 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:15:49.397663 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:15:49.398061 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:15:49.398064 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:15:49.398066 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:15:49.398479 (+ 413us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:15:49.398738 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0620 20:15:49.399075 (+ 337us) server_negotiation.cc:299] Negotiation successful
0620 20:15:49.399247 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":352,"thread_start_us":246,"threads_started":1}
W20260620 20:15:49.400480 18717 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48564
W20260620 20:15:49.401160 17252 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu pid 18587: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 20:15:49.401364 17252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu with pid 18587
I20260620 20:15:49.424525 17252 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu pid 18726
Jun 20 20:15:49 dist-test-slave-2hpg krb5kdc[18330](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986548, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.16.217.3@KRBTEST.COM
I20260620 20:15:49.443210 18928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:49.425452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60252 (local address 127.16.217.3:42419)
0620 20:15:49.425672 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0620 20:15:49.425678 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:15:49.425698 (+ 20us) server_negotiation.cc:407] Connection header received
0620 20:15:49.425952 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:15:49.425957 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:15:49.426031 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:15:49.426142 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:15:49.427754 (+ 1612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:49.428564 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:49.430356 (+ 1792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:49.430898 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:49.431012 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:15:49.436233 (+ 5221us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:15:49.436251 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:15:49.436266 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:15:49.436323 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:15:49.439939 (+ 3616us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:15:49.440726 (+ 787us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:15:49.440739 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:15:49.440743 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:15:49.440947 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:15:49.441380 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:15:49.441392 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:15:49.441396 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:15:49.441920 (+ 524us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:15:49.442287 (+ 367us) server_negotiation.cc:1035] Waiting for connection context
0620 20:15:49.442623 (+ 336us) server_negotiation.cc:299] Negotiation successful
0620 20:15:49.442849 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
W20260620 20:15:49.443969 18856 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60252
W20260620 20:15:49.444686 17252 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu pid 18726: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 20:15:49.444896 17252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu with pid 18726
I20260620 20:15:49.468915 17252 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu pid 18343
I20260620 20:15:49.470160 18415 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:15:49.582679 17252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu with pid 18343
2026-06-20T20:15:49Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (4832 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986536687152-17252-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 20:15:49 dist-test-slave-2hpg krb5kdc[18959](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 20:15:49 dist-test-slave-2hpg krb5kdc[18959](info): set up 2 sockets
Jun 20 20:15:49 dist-test-slave-2hpg krb5kdc[18959](info): commencing operation
krb5kdc: starting...
W20260620 20:15:50.662431 17252 mini_kdc.cc:121] Time spent starting KDC: real 1.036s user 0.004s sys 0.012s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986536687152-17252-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986536687152-17252-0/krb5kdc/test-user.keytab.
Jun 20 20:15:50 dist-test-slave-2hpg krb5kdc[18959](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986550, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T20:15:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T20:15:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.217.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.217.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.217.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.62.keytab.
Entry for principal kudu/127.16.217.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.62.keytab.
I20260620 20:15:50.847703 17252 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
/tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986536687152-17252-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986536687152-17252-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986536687152-17252-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986536687152-17252-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.217.62:38007
--webserver_interface=127.16.217.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.62.keytab
--principal=kudu/127.16.217.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.217.20:35029
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.217.62:38007
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986536687152-17252-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986536687152-17252-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986536687152-17252-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:15:51.167258 18972 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:15:51.167626 18972 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:15:51.167795 18972 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:15:51.175205 18972 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:15:51.175361 18972 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260620 20:15:51.175468 18972 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:15:51.175577 18972 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:15:51.175660 18972 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:15:51.175751 18972 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:15:51.185720 18972 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.217.20:35029
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986536687152-17252-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986536687152-17252-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.217.62:38007
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.62.keytab
--principal=kudu/127.16.217.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.217.62:38007
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986536687152-17252-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.217.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986536687152-17252-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:15:51.187044 18972 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:15:51.188382 18972 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:15:51.202225 18977 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:15:51.202179 18978 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:15:51.202250 18980 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:15:51.203295 18972 server_base.cc:1034] running on GCE node
I20260620 20:15:51.204093 18972 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:15:51.205529 18972 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:15:51.206739 18972 hybrid_clock.cc:648] HybridClock initialized: now 1781986551206685 us; error 29 us; skew 500 ppm
I20260620 20:15:51.207123 18972 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:15:51 dist-test-slave-2hpg krb5kdc[18959](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986551, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.217.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:15:51.210363 18972 init.cc:377] Logged in from keytab as kudu/127.16.217.62@KRBTEST.COM (short username kudu)
I20260620 20:15:51.212039 18972 webserver.cc:460] Webserver started at http://127.16.217.62:43231/ using document root <none> and password file <none>
I20260620 20:15:51.212709 18972 fs_manager.cc:359] Metadata directory not provided
I20260620 20:15:51.212793 18972 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:15:51.213048 18972 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:15:51.215960 18972 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986536687152-17252-0/minicluster-data/master-0/data/instance:
uuid: "ebfe81c8d02342e6bc65b45342cd2b9d"
format_stamp: "Formatted at 2026-06-20 20:15:51 on dist-test-slave-2hpg"
server_key: "8317e262011a621af20fe259315f9baf"
server_key_iv: "6247be39ea95a0e967a19f03615bd182"
server_key_version: "encryptionkey@0"
I20260620 20:15:51.216670 18972 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986536687152-17252-0/minicluster-data/master-0/wal/instance:
uuid: "ebfe81c8d02342e6bc65b45342cd2b9d"
format_stamp: "Formatted at 2026-06-20 20:15:51 on dist-test-slave-2hpg"
server_key: "8317e262011a621af20fe259315f9baf"
server_key_iv: "6247be39ea95a0e967a19f03615bd182"
server_key_version: "encryptionkey@0"
I20260620 20:15:51.222193 18972 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260620 20:15:51.225665 18987 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:15:51.227267 18972 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 20:15:51.227474 18972 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986536687152-17252-0/minicluster-data/master-0/data,/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986536687152-17252-0/minicluster-data/master-0/wal
uuid: "ebfe81c8d02342e6bc65b45342cd2b9d"
format_stamp: "Formatted at 2026-06-20 20:15:51 on dist-test-slave-2hpg"
server_key: "8317e262011a621af20fe259315f9baf"
server_key_iv: "6247be39ea95a0e967a19f03615bd182"
server_key_version: "encryptionkey@0"
I20260620 20:15:51.227696 18972 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986536687152-17252-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986536687152-17252-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986536687152-17252-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:15:51.257129 18972 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:15:51.261343 18972 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:15:51.261641 18972 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:15:51.284157 18972 rpc_server.cc:307] RPC server started. Bound to: 127.16.217.62:38007
I20260620 20:15:51.284176 19048 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.217.62:38007 every 8 connection(s)
I20260620 20:15:51.285614 18972 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986536687152-17252-0/minicluster-data/master-0/data/info.pb
I20260620 20:15:51.289686 19049 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:15:51.295459 17252 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu as pid 18972
I20260620 20:15:51.295694 17252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986536687152-17252-0/minicluster-data/master-0/wal/instance
I20260620 20:15:51.296350 17252 external_mini_cluster.cc:1428] Setting key a93dc8482b304830d825c8731b75b185
I20260620 20:15:51.302284 19049 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ebfe81c8d02342e6bc65b45342cd2b9d: Bootstrap starting.
I20260620 20:15:51.306270 19049 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ebfe81c8d02342e6bc65b45342cd2b9d: Neither blocks nor log segments found. Creating new log.
I20260620 20:15:51.307420 19049 log.cc:824] T 00000000000000000000000000000000 P ebfe81c8d02342e6bc65b45342cd2b9d: Log is configured to *not* fsync() on all Append() calls
I20260620 20:15:51.310570 19049 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ebfe81c8d02342e6bc65b45342cd2b9d: No bootstrap required, opened a new log
Jun 20 20:15:51 dist-test-slave-2hpg krb5kdc[18959](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986550, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.217.62@KRBTEST.COM
I20260620 20:15:51.321513 19049 raft_consensus.cc:348] T 00000000000000000000000000000000 P ebfe81c8d02342e6bc65b45342cd2b9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebfe81c8d02342e6bc65b45342cd2b9d" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 38007 } }
I20260620 20:15:51.322093 19049 raft_consensus.cc:374] T 00000000000000000000000000000000 P ebfe81c8d02342e6bc65b45342cd2b9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:15:51.322217 19049 raft_consensus.cc:729] T 00000000000000000000000000000000 P ebfe81c8d02342e6bc65b45342cd2b9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ebfe81c8d02342e6bc65b45342cd2b9d, State: Initialized, Role: FOLLOWER
I20260620 20:15:51.322667 19052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:51.298569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55262 (local address 127.16.217.62:38007)
0620 20:15:51.299606 (+ 1037us) server_negotiation.cc:206] Beginning negotiation
0620 20:15:51.299623 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 20:15:51.299679 (+ 56us) server_negotiation.cc:407] Connection header received
0620 20:15:51.302206 (+ 2527us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:15:51.302234 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:15:51.302759 (+ 525us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:15:51.303229 (+ 470us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:15:51.304536 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:51.306537 (+ 2001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:51.307982 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:51.308878 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:51.309084 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:15:51.313099 (+ 4015us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:15:51.313110 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:15:51.313134 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:15:51.313194 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:15:51.317623 (+ 4429us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:15:51.318398 (+ 775us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:15:51.318408 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:15:51.318417 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:15:51.318651 (+ 234us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:15:51.319077 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:15:51.319083 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:15:51.319088 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:15:51.319905 (+ 817us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:15:51.320219 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0620 20:15:51.320643 (+ 424us) server_negotiation.cc:299] Negotiation successful
0620 20:15:51.321088 (+ 445us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":756,"thread_start_us":467,"threads_started":1}
I20260620 20:15:51.322978 19049 consensus_queue.cc:260] T 00000000000000000000000000000000 P ebfe81c8d02342e6bc65b45342cd2b9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebfe81c8d02342e6bc65b45342cd2b9d" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 38007 } }
I20260620 20:15:51.323290 19049 raft_consensus.cc:388] T 00000000000000000000000000000000 P ebfe81c8d02342e6bc65b45342cd2b9d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 20:15:51.323453 19049 raft_consensus.cc:482] T 00000000000000000000000000000000 P ebfe81c8d02342e6bc65b45342cd2b9d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 20:15:51.323671 19049 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ebfe81c8d02342e6bc65b45342cd2b9d [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:15:51.326390 19049 raft_consensus.cc:504] T 00000000000000000000000000000000 P ebfe81c8d02342e6bc65b45342cd2b9d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebfe81c8d02342e6bc65b45342cd2b9d" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 38007 } }
I20260620 20:15:51.326930 19049 leader_election.cc:302] T 00000000000000000000000000000000 P ebfe81c8d02342e6bc65b45342cd2b9d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ebfe81c8d02342e6bc65b45342cd2b9d; no voters:
I20260620 20:15:51.327535 19049 leader_election.cc:288] T 00000000000000000000000000000000 P ebfe81c8d02342e6bc65b45342cd2b9d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 20:15:51.327682 19054 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ebfe81c8d02342e6bc65b45342cd2b9d [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:15:51.328773 19054 raft_consensus.cc:686] T 00000000000000000000000000000000 P ebfe81c8d02342e6bc65b45342cd2b9d [term 1 LEADER]: Becoming Leader. State: Replica: ebfe81c8d02342e6bc65b45342cd2b9d, State: Running, Role: LEADER
I20260620 20:15:51.329357 19054 consensus_queue.cc:237] T 00000000000000000000000000000000 P ebfe81c8d02342e6bc65b45342cd2b9d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebfe81c8d02342e6bc65b45342cd2b9d" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 38007 } }
I20260620 20:15:51.330314 19049 sys_catalog.cc:564] T 00000000000000000000000000000000 P ebfe81c8d02342e6bc65b45342cd2b9d [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:15:51.332356 19055 sys_catalog.cc:455] T 00000000000000000000000000000000 P ebfe81c8d02342e6bc65b45342cd2b9d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ebfe81c8d02342e6bc65b45342cd2b9d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebfe81c8d02342e6bc65b45342cd2b9d" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 38007 } } }
I20260620 20:15:51.332826 19055 sys_catalog.cc:458] T 00000000000000000000000000000000 P ebfe81c8d02342e6bc65b45342cd2b9d [sys.catalog]: This master's current role is: LEADER
I20260620 20:15:51.332463 19056 sys_catalog.cc:455] T 00000000000000000000000000000000 P ebfe81c8d02342e6bc65b45342cd2b9d [sys.catalog]: SysCatalogTable state changed. Reason: New leader ebfe81c8d02342e6bc65b45342cd2b9d. Latest consensus state: current_term: 1 leader_uuid: "ebfe81c8d02342e6bc65b45342cd2b9d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebfe81c8d02342e6bc65b45342cd2b9d" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 38007 } } }
I20260620 20:15:51.334460 19056 sys_catalog.cc:458] T 00000000000000000000000000000000 P ebfe81c8d02342e6bc65b45342cd2b9d [sys.catalog]: This master's current role is: LEADER
I20260620 20:15:51.339964 19059 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 20:15:51.346654 19059 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 20:15:51.355885 19059 catalog_manager.cc:1347] Generated new cluster ID: 2d20ec2a28474961af702afccbf4878e
I20260620 20:15:51.356019 19059 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 20:15:51.385739 19059 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 20:15:51.387077 19059 catalog_manager.cc:1504] Loading token signing keys...
I20260620 20:15:51.395447 19059 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ebfe81c8d02342e6bc65b45342cd2b9d: Generated new TSK 0
I20260620 20:15:51.396441 19059 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.217.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.217.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.217.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.1.keytab.
Entry for principal kudu/127.16.217.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.1.keytab.
I20260620 20:15:51.449029 17252 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
/tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986536687152-17252-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986536687152-17252-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986536687152-17252-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986536687152-17252-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.217.1:0
--local_ip_for_outbound_sockets=127.16.217.1
--webserver_interface=127.16.217.1
--webserver_port=0
--tserver_master_addrs=127.16.217.62:38007
--keytab_file=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.1.keytab
--principal=kudu/127.16.217.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.217.20:35029
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986536687152-17252-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986536687152-17252-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986536687152-17252-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:15:51.781427 19075 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:15:51.781795 19075 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:15:51.781999 19075 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:15:51.789274 19075 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:15:51.789429 19075 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:15:51.789589 19075 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260620 20:15:51.789683 19075 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 20:15:51.789834 19075 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.217.1
I20260620 20:15:51.799774 19075 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.217.20:35029
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986536687152-17252-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986536687152-17252-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.1.keytab
--principal=kudu/127.16.217.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.217.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986536687152-17252-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.217.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.16.217.62:38007
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.217.1
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986536687152-17252-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:15:51.801158 19075 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:15:51.802599 19075 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:15:51.817514 19080 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:15:51.817471 19083 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:15:51.817514 19081 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:15:51.818332 19075 server_base.cc:1034] running on GCE node
I20260620 20:15:51.819036 19075 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:15:51.819967 19075 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:15:51.821199 19075 hybrid_clock.cc:648] HybridClock initialized: now 1781986551821156 us; error 54 us; skew 500 ppm
I20260620 20:15:51.821528 19075 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:15:51 dist-test-slave-2hpg krb5kdc[18959](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986551, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.217.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:15:51.824658 19075 init.cc:377] Logged in from keytab as kudu/127.16.217.1@KRBTEST.COM (short username kudu)
I20260620 20:15:51.826663 19075 webserver.cc:460] Webserver started at http://127.16.217.1:44049/ using document root <none> and password file <none>
I20260620 20:15:51.827378 19075 fs_manager.cc:359] Metadata directory not provided
I20260620 20:15:51.827546 19075 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:15:51.827898 19075 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:15:51.831202 19075 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986536687152-17252-0/minicluster-data/ts-0/data/instance:
uuid: "426541981bf84ca1a3e874674a81d687"
format_stamp: "Formatted at 2026-06-20 20:15:51 on dist-test-slave-2hpg"
server_key: "39efd96b9184ac78724b0737949d70ca"
server_key_iv: "8715180773f005cfab526cba2cff9519"
server_key_version: "encryptionkey@0"
I20260620 20:15:51.832149 19075 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986536687152-17252-0/minicluster-data/ts-0/wal/instance:
uuid: "426541981bf84ca1a3e874674a81d687"
format_stamp: "Formatted at 2026-06-20 20:15:51 on dist-test-slave-2hpg"
server_key: "39efd96b9184ac78724b0737949d70ca"
server_key_iv: "8715180773f005cfab526cba2cff9519"
server_key_version: "encryptionkey@0"
I20260620 20:15:51.837455 19075 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 20:15:51.840883 19090 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:15:51.842195 19075 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260620 20:15:51.842401 19075 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986536687152-17252-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986536687152-17252-0/minicluster-data/ts-0/wal
uuid: "426541981bf84ca1a3e874674a81d687"
format_stamp: "Formatted at 2026-06-20 20:15:51 on dist-test-slave-2hpg"
server_key: "39efd96b9184ac78724b0737949d70ca"
server_key_iv: "8715180773f005cfab526cba2cff9519"
server_key_version: "encryptionkey@0"
I20260620 20:15:51.842598 19075 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986536687152-17252-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986536687152-17252-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986536687152-17252-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:15:51.869220 19075 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:15:51.873121 19075 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:15:51.873410 19075 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:15:51.875979 19075 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:15:51.876120 19075 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:15:51.876255 19075 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:15:51.876355 19075 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 20:15:51 dist-test-slave-2hpg krb5kdc[18959](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986551, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.217.1@KRBTEST.COM for kudu/127.16.217.62@KRBTEST.COM
I20260620 20:15:51.900542 19099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:51.876735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.217.1:59403 (local address 127.16.217.62:38007)
0620 20:15:51.877280 (+ 545us) server_negotiation.cc:206] Beginning negotiation
0620 20:15:51.877287 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:15:51.878561 (+ 1274us) server_negotiation.cc:407] Connection header received
0620 20:15:51.880585 (+ 2024us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:15:51.880591 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:15:51.880672 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:15:51.880779 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:15:51.883233 (+ 2454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:51.884116 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:51.887455 (+ 3339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:51.888038 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:51.888192 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:15:51.893740 (+ 5548us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:15:51.893748 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:15:51.893755 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:15:51.893798 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:15:51.896850 (+ 3052us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:15:51.898015 (+ 1165us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:15:51.898028 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:15:51.898034 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:15:51.898170 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:15:51.898779 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:15:51.898785 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:15:51.898789 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:15:51.899101 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:15:51.899258 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0620 20:15:51.899968 (+ 710us) server_negotiation.cc:299] Negotiation successful
0620 20:15:51.900178 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":402,"thread_start_us":296,"threads_started":1}
I20260620 20:15:51.901578 19100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:51.877432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:38007 (local address 127.16.217.1:59403)
0620 20:15:51.878376 (+ 944us) negotiation.cc:107] Waiting for socket to connect
0620 20:15:51.878419 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 20:15:51.880292 (+ 1873us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:15:51.881192 (+ 900us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:15:51.881203 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:15:51.882075 (+ 872us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:15:51.883066 (+ 991us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:15:51.883085 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:15:51.884307 (+ 1222us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:15:51.884316 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:15:51.887276 (+ 2960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:15:51.887296 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:15:51.888400 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:15:51.888407 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:15:51.888821 (+ 414us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:15:51.889677 (+ 856us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:15:51.889717 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:15:51.893444 (+ 3727us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:15:51.896999 (+ 3555us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:15:51.897013 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:15:51.897037 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:15:51.897834 (+ 797us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:15:51.898288 (+ 454us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:15:51.898296 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:15:51.898303 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:15:51.898626 (+ 323us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:15:51.899243 (+ 617us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:15:51.899250 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:15:51.899613 (+ 363us) client_negotiation.cc:769] Sending connection context
0620 20:15:51.899881 (+ 268us) client_negotiation.cc:240] Negotiation successful
0620 20:15:51.900387 (+ 506us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":673,"thread_start_us":427,"threads_started":1}
I20260620 20:15:51.923496 19171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:51.908364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:38007 (local address 127.16.217.1:38643)
0620 20:15:51.909117 (+ 753us) negotiation.cc:107] Waiting for socket to connect
0620 20:15:51.909142 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0620 20:15:51.909306 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:15:51.909623 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:15:51.909630 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:15:51.910416 (+ 786us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:15:51.910979 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:15:51.910993 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:15:51.912016 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:15:51.912024 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:15:51.914171 (+ 2147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:15:51.914190 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:15:51.915223 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:15:51.915232 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:15:51.915471 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:15:51.916228 (+ 757us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:15:51.916258 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:15:51.917203 (+ 945us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:15:51.920549 (+ 3346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:15:51.920554 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:15:51.920558 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:15:51.921097 (+ 539us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:15:51.921489 (+ 392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:15:51.921495 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:15:51.921499 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:15:51.921638 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:15:51.922214 (+ 576us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:15:51.922222 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:15:51.922409 (+ 187us) client_negotiation.cc:769] Sending connection context
0620 20:15:51.922576 (+ 167us) client_negotiation.cc:240] Negotiation successful
0620 20:15:51.922952 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":562,"thread_start_us":383,"threads_started":1}
I20260620 20:15:51.924286 19099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:51.908359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.217.1:38643 (local address 127.16.217.62:38007)
0620 20:15:51.908565 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0620 20:15:51.908584 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0620 20:15:51.909266 (+ 682us) server_negotiation.cc:407] Connection header received
0620 20:15:51.909423 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:15:51.909426 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:15:51.909484 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:15:51.909565 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:15:51.911160 (+ 1595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:51.911866 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:51.914405 (+ 2539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:51.914982 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:51.915122 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:15:51.917402 (+ 2280us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:15:51.917407 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:15:51.917410 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:15:51.917457 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:15:51.920415 (+ 2958us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:15:51.921232 (+ 817us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:15:51.921239 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:15:51.921244 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:15:51.921383 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:15:51.921753 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:15:51.921760 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:15:51.921763 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:15:51.922099 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:15:51.922246 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0620 20:15:51.923840 (+ 1594us) server_negotiation.cc:299] Negotiation successful
0620 20:15:51.924068 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 20:15:51.934028 19075 rpc_server.cc:307] RPC server started. Bound to: 127.16.217.1:40783
I20260620 20:15:51.934113 19211 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.217.1:40783 every 8 connection(s)
I20260620 20:15:51.935225 19075 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986536687152-17252-0/minicluster-data/ts-0/data/info.pb
I20260620 20:15:51.937253 19212 heartbeater.cc:347] Connected to a master server at 127.16.217.62:38007
I20260620 20:15:51.937484 19212 heartbeater.cc:464] Registering TS with master...
I20260620 20:15:51.938143 19212 heartbeater.cc:511] Master 127.16.217.62:38007 requested a full tablet report, sending...
I20260620 20:15:51.938696 17252 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu as pid 19075
I20260620 20:15:51.938913 17252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986536687152-17252-0/minicluster-data/ts-0/wal/instance
I20260620 20:15:51.939548 17252 external_mini_cluster.cc:1428] Setting key 13c5f341bbae865258612d1dbeb75ae0
I20260620 20:15:51.940035 19004 ts_manager.cc:194] Registered new tserver with Master: 426541981bf84ca1a3e874674a81d687 (127.16.217.1:40783)
I20260620 20:15:51.941694 19004 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.217.1@KRBTEST.COM'} at 127.16.217.1:59403
WARNING: no policy specified for kudu/127.16.217.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.217.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.217.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.2.keytab.
Entry for principal kudu/127.16.217.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.2.keytab.
I20260620 20:15:51.978441 17252 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
/tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986536687152-17252-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986536687152-17252-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986536687152-17252-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986536687152-17252-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.217.2:0
--local_ip_for_outbound_sockets=127.16.217.2
--webserver_interface=127.16.217.2
--webserver_port=0
--tserver_master_addrs=127.16.217.62:38007
--keytab_file=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.2.keytab
--principal=kudu/127.16.217.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.217.20:35029
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986536687152-17252-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986536687152-17252-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986536687152-17252-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:15:52.294294 19216 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:15:52.294700 19216 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:15:52.294870 19216 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:15:52.306406 19216 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:15:52.306536 19216 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:15:52.306661 19216 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260620 20:15:52.306727 19216 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 20:15:52.306823 19216 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.217.2
I20260620 20:15:52.316847 19216 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.217.20:35029
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986536687152-17252-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986536687152-17252-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.2.keytab
--principal=kudu/127.16.217.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.217.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986536687152-17252-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.217.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.16.217.62:38007
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.217.2
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986536687152-17252-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:15:52.318135 19216 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:15:52.319415 19216 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:15:52.332425 19222 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:15:52.332481 19221 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:15:52.332508 19224 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:15:52.333637 19216 server_base.cc:1034] running on GCE node
I20260620 20:15:52.334455 19216 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:15:52.335428 19216 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:15:52.336651 19216 hybrid_clock.cc:648] HybridClock initialized: now 1781986552336602 us; error 66 us; skew 500 ppm
I20260620 20:15:52.336961 19216 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:15:52 dist-test-slave-2hpg krb5kdc[18959](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986552, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.217.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:15:52.340103 19216 init.cc:377] Logged in from keytab as kudu/127.16.217.2@KRBTEST.COM (short username kudu)
I20260620 20:15:52.341513 19216 webserver.cc:460] Webserver started at http://127.16.217.2:40879/ using document root <none> and password file <none>
I20260620 20:15:52.342278 19216 fs_manager.cc:359] Metadata directory not provided
I20260620 20:15:52.342463 19216 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:15:52.342801 19216 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:15:52.345607 19216 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986536687152-17252-0/minicluster-data/ts-1/data/instance:
uuid: "ce1bf2b4620340b88840348ea13eb73d"
format_stamp: "Formatted at 2026-06-20 20:15:52 on dist-test-slave-2hpg"
server_key: "de3d06af04046a50bb38047d6eba4f16"
server_key_iv: "a5726e7f2fe16717d05fb71fb6788680"
server_key_version: "encryptionkey@0"
I20260620 20:15:52.346392 19216 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986536687152-17252-0/minicluster-data/ts-1/wal/instance:
uuid: "ce1bf2b4620340b88840348ea13eb73d"
format_stamp: "Formatted at 2026-06-20 20:15:52 on dist-test-slave-2hpg"
server_key: "de3d06af04046a50bb38047d6eba4f16"
server_key_iv: "a5726e7f2fe16717d05fb71fb6788680"
server_key_version: "encryptionkey@0"
I20260620 20:15:52.351250 19216 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260620 20:15:52.354537 19231 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:15:52.355846 19216 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 20:15:52.356070 19216 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986536687152-17252-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986536687152-17252-0/minicluster-data/ts-1/wal
uuid: "ce1bf2b4620340b88840348ea13eb73d"
format_stamp: "Formatted at 2026-06-20 20:15:52 on dist-test-slave-2hpg"
server_key: "de3d06af04046a50bb38047d6eba4f16"
server_key_iv: "a5726e7f2fe16717d05fb71fb6788680"
server_key_version: "encryptionkey@0"
I20260620 20:15:52.356271 19216 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986536687152-17252-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986536687152-17252-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986536687152-17252-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:15:52.379372 19216 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:15:52.384048 19216 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:15:52.384370 19216 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:15:52.387244 19216 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:15:52.387413 19216 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:15:52.387538 19216 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:15:52.387631 19216 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 20:15:52 dist-test-slave-2hpg krb5kdc[18959](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986552, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.217.2@KRBTEST.COM for kudu/127.16.217.62@KRBTEST.COM
I20260620 20:15:52.412354 19240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:52.388109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:38007 (local address 127.16.217.2:60747)
0620 20:15:52.389018 (+ 909us) negotiation.cc:107] Waiting for socket to connect
0620 20:15:52.389063 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 20:15:52.390946 (+ 1883us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:15:52.391682 (+ 736us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:15:52.391690 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:15:52.392377 (+ 687us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:15:52.393194 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:15:52.393212 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:15:52.394612 (+ 1400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:15:52.394622 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:15:52.398028 (+ 3406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:15:52.398049 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:15:52.399019 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:15:52.399031 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:15:52.399407 (+ 376us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:15:52.400235 (+ 828us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:15:52.400261 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:15:52.403138 (+ 2877us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:15:52.407020 (+ 3882us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:15:52.407033 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:15:52.407053 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:15:52.407759 (+ 706us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:15:52.408346 (+ 587us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:15:52.408352 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:15:52.408357 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:15:52.408666 (+ 309us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:15:52.409397 (+ 731us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:15:52.409405 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:15:52.410021 (+ 616us) client_negotiation.cc:769] Sending connection context
0620 20:15:52.410412 (+ 391us) client_negotiation.cc:240] Negotiation successful
0620 20:15:52.410984 (+ 572us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":583,"thread_start_us":348,"threads_started":1}
I20260620 20:15:52.413393 19099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:52.387599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.217.2:60747 (local address 127.16.217.62:38007)
0620 20:15:52.387797 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0620 20:15:52.387803 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:15:52.389216 (+ 1413us) server_negotiation.cc:407] Connection header received
0620 20:15:52.391249 (+ 2033us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:15:52.391254 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:15:52.391348 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:15:52.391459 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:15:52.393372 (+ 1913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:52.394414 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:52.398208 (+ 3794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:52.398747 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:52.398888 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:15:52.403487 (+ 4599us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:15:52.403495 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:15:52.403502 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:15:52.403570 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:15:52.406867 (+ 3297us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:15:52.408008 (+ 1141us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:15:52.408014 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:15:52.408019 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:15:52.408188 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:15:52.408799 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:15:52.408804 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:15:52.408807 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:15:52.409248 (+ 441us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:15:52.409428 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0620 20:15:52.412992 (+ 3564us) server_negotiation.cc:299] Negotiation successful
0620 20:15:52.413166 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260620 20:15:52.437325 19216 rpc_server.cc:307] RPC server started. Bound to: 127.16.217.2:40071
I20260620 20:15:52.437356 19350 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.217.2:40071 every 8 connection(s)
I20260620 20:15:52.438464 19099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:52.420861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.217.2:44393 (local address 127.16.217.62:38007)
0620 20:15:52.421031 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0620 20:15:52.421035 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:15:52.421696 (+ 661us) server_negotiation.cc:407] Connection header received
0620 20:15:52.422032 (+ 336us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:15:52.422041 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:15:52.422131 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:15:52.422208 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:15:52.423989 (+ 1781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:52.424936 (+ 947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:52.427429 (+ 2493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:52.428177 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:52.428288 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:15:52.430852 (+ 2564us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:15:52.430858 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:15:52.430861 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:15:52.430893 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:15:52.435240 (+ 4347us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:15:52.436196 (+ 956us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:15:52.436205 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:15:52.436211 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:15:52.436394 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:15:52.436911 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:15:52.436919 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:15:52.436923 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:15:52.437380 (+ 457us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:15:52.437526 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0620 20:15:52.438072 (+ 546us) server_negotiation.cc:299] Negotiation successful
0620 20:15:52.438243 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 20:15:52.438738 19315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:52.420968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:38007 (local address 127.16.217.2:44393)
0620 20:15:52.421587 (+ 619us) negotiation.cc:107] Waiting for socket to connect
0620 20:15:52.421603 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 20:15:52.421769 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:15:52.422474 (+ 705us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:15:52.422481 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:15:52.423242 (+ 761us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:15:52.423788 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:15:52.423805 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:15:52.425099 (+ 1294us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:15:52.425108 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:15:52.427279 (+ 2171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:15:52.427292 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:15:52.428384 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:15:52.428390 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:15:52.428583 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:15:52.429717 (+ 1134us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:15:52.429746 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:15:52.430646 (+ 900us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:15:52.435397 (+ 4751us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:15:52.435405 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:15:52.435411 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:15:52.436020 (+ 609us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:15:52.436548 (+ 528us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:15:52.436556 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:15:52.436567 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:15:52.436759 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:15:52.437584 (+ 825us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:15:52.437589 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:15:52.437774 (+ 185us) client_negotiation.cc:769] Sending connection context
0620 20:15:52.437929 (+ 155us) client_negotiation.cc:240] Negotiation successful
0620 20:15:52.438255 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":487,"thread_start_us":350,"threads_started":1}
I20260620 20:15:52.439049 19216 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986536687152-17252-0/minicluster-data/ts-1/data/info.pb
I20260620 20:15:52.442056 19351 heartbeater.cc:347] Connected to a master server at 127.16.217.62:38007
I20260620 20:15:52.442309 19351 heartbeater.cc:464] Registering TS with master...
I20260620 20:15:52.443150 19351 heartbeater.cc:511] Master 127.16.217.62:38007 requested a full tablet report, sending...
I20260620 20:15:52.444403 19004 ts_manager.cc:194] Registered new tserver with Master: ce1bf2b4620340b88840348ea13eb73d (127.16.217.2:40071)
I20260620 20:15:52.445199 19004 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.217.2@KRBTEST.COM'} at 127.16.217.2:60747
I20260620 20:15:52.445168 17252 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu as pid 19216
I20260620 20:15:52.445370 17252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986536687152-17252-0/minicluster-data/ts-1/wal/instance
I20260620 20:15:52.445911 17252 external_mini_cluster.cc:1428] Setting key f4172c852e2e407a91122e574490653c
WARNING: no policy specified for kudu/127.16.217.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.217.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.217.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.3.keytab.
Entry for principal kudu/127.16.217.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.3.keytab.
I20260620 20:15:52.485723 17252 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
/tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986536687152-17252-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986536687152-17252-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986536687152-17252-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986536687152-17252-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.217.3:0
--local_ip_for_outbound_sockets=127.16.217.3
--webserver_interface=127.16.217.3
--webserver_port=0
--tserver_master_addrs=127.16.217.62:38007
--keytab_file=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.3.keytab
--principal=kudu/127.16.217.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.217.20:35029
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986536687152-17252-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986536687152-17252-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986536687152-17252-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:15:52.802018 19355 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:15:52.802345 19355 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:15:52.802521 19355 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:15:52.809763 19355 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:15:52.809928 19355 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:15:52.810079 19355 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260620 20:15:52.810170 19355 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 20:15:52.810309 19355 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.217.3
I20260620 20:15:52.821079 19355 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.217.20:35029
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986536687152-17252-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986536687152-17252-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.3.keytab
--principal=kudu/127.16.217.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.217.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986536687152-17252-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.217.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.16.217.62:38007
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.217.3
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986536687152-17252-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:15:52.822433 19355 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:15:52.823705 19355 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:15:52.836743 19360 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:15:52.836730 19363 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:15:52.836784 19361 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:15:52.837532 19355 server_base.cc:1034] running on GCE node
I20260620 20:15:52.838172 19355 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:15:52.838995 19355 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:15:52.840245 19355 hybrid_clock.cc:648] HybridClock initialized: now 1781986552840198 us; error 56 us; skew 500 ppm
I20260620 20:15:52.840605 19355 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:15:52 dist-test-slave-2hpg krb5kdc[18959](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986552, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.217.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:15:52.843758 19355 init.cc:377] Logged in from keytab as kudu/127.16.217.3@KRBTEST.COM (short username kudu)
I20260620 20:15:52.845108 19355 webserver.cc:460] Webserver started at http://127.16.217.3:35913/ using document root <none> and password file <none>
I20260620 20:15:52.845846 19355 fs_manager.cc:359] Metadata directory not provided
I20260620 20:15:52.846040 19355 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:15:52.846343 19355 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:15:52.849157 19355 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986536687152-17252-0/minicluster-data/ts-2/data/instance:
uuid: "56cd29ec76f24959a145a593ab44b2d0"
format_stamp: "Formatted at 2026-06-20 20:15:52 on dist-test-slave-2hpg"
server_key: "fb51f4d6753625ed19f96a5dfda436cd"
server_key_iv: "0185fbe24388470a820dc7b77fea4989"
server_key_version: "encryptionkey@0"
I20260620 20:15:52.849993 19355 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986536687152-17252-0/minicluster-data/ts-2/wal/instance:
uuid: "56cd29ec76f24959a145a593ab44b2d0"
format_stamp: "Formatted at 2026-06-20 20:15:52 on dist-test-slave-2hpg"
server_key: "fb51f4d6753625ed19f96a5dfda436cd"
server_key_iv: "0185fbe24388470a820dc7b77fea4989"
server_key_version: "encryptionkey@0"
I20260620 20:15:52.855120 19355 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260620 20:15:52.858346 19370 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:15:52.859674 19355 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 20:15:52.859885 19355 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986536687152-17252-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986536687152-17252-0/minicluster-data/ts-2/wal
uuid: "56cd29ec76f24959a145a593ab44b2d0"
format_stamp: "Formatted at 2026-06-20 20:15:52 on dist-test-slave-2hpg"
server_key: "fb51f4d6753625ed19f96a5dfda436cd"
server_key_iv: "0185fbe24388470a820dc7b77fea4989"
server_key_version: "encryptionkey@0"
I20260620 20:15:52.860083 19355 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986536687152-17252-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986536687152-17252-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986536687152-17252-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:15:52.879508 19355 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:15:52.883358 19355 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:15:52.883648 19355 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:15:52.886327 19355 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:15:52.886479 19355 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:15:52.886649 19355 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:15:52.886739 19355 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 20:15:52 dist-test-slave-2hpg krb5kdc[18959](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986552, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.217.3@KRBTEST.COM for kudu/127.16.217.62@KRBTEST.COM
I20260620 20:15:52.910132 19379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:52.887867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:38007 (local address 127.16.217.3:46279)
0620 20:15:52.888662 (+ 795us) negotiation.cc:107] Waiting for socket to connect
0620 20:15:52.888701 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 20:15:52.890574 (+ 1873us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:15:52.891346 (+ 772us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:15:52.891366 (+ 20us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:15:52.892286 (+ 920us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:15:52.893434 (+ 1148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:15:52.893467 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:15:52.894908 (+ 1441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:15:52.894912 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:15:52.897067 (+ 2155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:15:52.897079 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:15:52.898097 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:15:52.898105 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:15:52.898303 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:15:52.898962 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:15:52.898983 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:15:52.901382 (+ 2399us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:15:52.905112 (+ 3730us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:15:52.905121 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:15:52.905143 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:15:52.905627 (+ 484us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:15:52.906170 (+ 543us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:15:52.906174 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:15:52.906179 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:15:52.906457 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:15:52.907227 (+ 770us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:15:52.907236 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:15:52.907563 (+ 327us) client_negotiation.cc:769] Sending connection context
0620 20:15:52.907869 (+ 306us) client_negotiation.cc:240] Negotiation successful
0620 20:15:52.908405 (+ 536us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":557,"thread_start_us":343,"threads_started":1}
I20260620 20:15:52.910650 19099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:52.887262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.217.3:46279 (local address 127.16.217.62:38007)
0620 20:15:52.887456 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0620 20:15:52.887460 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:15:52.888836 (+ 1376us) server_negotiation.cc:407] Connection header received
0620 20:15:52.890843 (+ 2007us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:15:52.890848 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:15:52.890937 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:15:52.891052 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:15:52.893667 (+ 2615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:52.894710 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:52.897210 (+ 2500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:52.897788 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:52.897976 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:15:52.901600 (+ 3624us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:15:52.901609 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:15:52.901616 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:15:52.901666 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:15:52.904988 (+ 3322us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:15:52.905796 (+ 808us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:15:52.905803 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:15:52.905806 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:15:52.906011 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:15:52.906582 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:15:52.906588 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:15:52.906591 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:15:52.907019 (+ 428us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:15:52.907154 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0620 20:15:52.910095 (+ 2941us) server_negotiation.cc:299] Negotiation successful
0620 20:15:52.910323 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260620 20:15:52.929986 19435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:52.915780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:38007 (local address 127.16.217.3:40191)
0620 20:15:52.916424 (+ 644us) negotiation.cc:107] Waiting for socket to connect
0620 20:15:52.916446 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 20:15:52.916569 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:15:52.916880 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:15:52.916884 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:15:52.917548 (+ 664us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:15:52.918030 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:15:52.918050 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:15:52.919435 (+ 1385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:15:52.919440 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:15:52.921029 (+ 1589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:15:52.921040 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:15:52.921984 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:15:52.921994 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:15:52.922257 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:15:52.922912 (+ 655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:15:52.922942 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:15:52.924238 (+ 1296us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:15:52.927055 (+ 2817us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:15:52.927063 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:15:52.927069 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:15:52.927773 (+ 704us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:15:52.928189 (+ 416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:15:52.928198 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:15:52.928202 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:15:52.928395 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:15:52.929111 (+ 716us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:15:52.929114 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:15:52.929254 (+ 140us) client_negotiation.cc:769] Sending connection context
0620 20:15:52.929365 (+ 111us) client_negotiation.cc:240] Negotiation successful
0620 20:15:52.929582 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":332,"threads_started":1}
I20260620 20:15:52.930279 19099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:52.915659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.217.3:40191 (local address 127.16.217.62:38007)
0620 20:15:52.915800 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0620 20:15:52.915804 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:15:52.916523 (+ 719us) server_negotiation.cc:407] Connection header received
0620 20:15:52.916692 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:15:52.916697 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:15:52.916761 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:15:52.916886 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:15:52.918224 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:52.919286 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:52.921174 (+ 1888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:52.921739 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:52.921931 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:15:52.924429 (+ 2498us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:15:52.924433 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:15:52.924436 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:15:52.924463 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:15:52.926917 (+ 2454us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:15:52.927909 (+ 992us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:15:52.927917 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:15:52.927920 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:15:52.928063 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:15:52.928534 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:15:52.928542 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:15:52.928548 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:15:52.928985 (+ 437us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:15:52.929170 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0620 20:15:52.929518 (+ 348us) server_negotiation.cc:299] Negotiation successful
0620 20:15:52.929815 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 20:15:52.944389 19212 heartbeater.cc:503] Master 127.16.217.62:38007 was elected leader, sending a full tablet report...
I20260620 20:15:52.944648 19355 rpc_server.cc:307] RPC server started. Bound to: 127.16.217.3:45785
I20260620 20:15:52.944698 19489 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.217.3:45785 every 8 connection(s)
I20260620 20:15:52.945921 19355 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986536687152-17252-0/minicluster-data/ts-2/data/info.pb
I20260620 20:15:52.947882 19490 heartbeater.cc:347] Connected to a master server at 127.16.217.62:38007
I20260620 20:15:52.948062 19490 heartbeater.cc:464] Registering TS with master...
I20260620 20:15:52.948611 19490 heartbeater.cc:511] Master 127.16.217.62:38007 requested a full tablet report, sending...
I20260620 20:15:52.949730 19004 ts_manager.cc:194] Registered new tserver with Master: 56cd29ec76f24959a145a593ab44b2d0 (127.16.217.3:45785)
I20260620 20:15:52.950548 19004 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.217.3@KRBTEST.COM'} at 127.16.217.3:46279
I20260620 20:15:52.952103 17252 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu as pid 19355
I20260620 20:15:52.952275 17252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986536687152-17252-0/minicluster-data/ts-2/wal/instance
I20260620 20:15:52.952693 17252 external_mini_cluster.cc:1428] Setting key d17bdefc5f1c0fc733d34077d78e1ce7
I20260620 20:15:52.955338 17252 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 20:15:52 dist-test-slave-2hpg krb5kdc[18959](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986552, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 20:15:52 dist-test-slave-2hpg krb5kdc[18959](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986552, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.217.62@KRBTEST.COM
I20260620 20:15:53.002110 19099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:52.988823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55264 (local address 127.16.217.62:38007)
0620 20:15:52.989065 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0620 20:15:52.989072 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:15:52.989460 (+ 388us) server_negotiation.cc:407] Connection header received
0620 20:15:52.989655 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:15:52.989667 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:15:52.989797 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:15:52.989926 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:15:52.991226 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:52.991978 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:52.993189 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:52.993636 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:52.993767 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:15:52.997459 (+ 3692us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:15:52.997464 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:15:52.997467 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:15:52.997502 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:15:52.999907 (+ 2405us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:15:53.000491 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:15:53.000495 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:15:53.000497 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:15:53.000622 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:15:53.000928 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:15:53.000931 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:15:53.000933 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:15:53.001165 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:15:53.001330 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0620 20:15:53.001611 (+ 281us) server_negotiation.cc:299] Negotiation successful
0620 20:15:53.001763 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
I20260620 20:15:53.006325 19004 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55264:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 20:15:53.013253 19004 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 20:15:53.038259 19504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:53.027468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.2:40071 (local address 127.0.0.1:45486)
0620 20:15:53.031275 (+ 3807us) negotiation.cc:107] Waiting for socket to connect
0620 20:15:53.031299 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 20:15:53.031420 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:15:53.031931 (+ 511us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:15:53.031935 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:15:53.031958 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:15:53.032335 (+ 377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:15:53.032344 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:15:53.034071 (+ 1727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:15:53.034076 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:15:53.035744 (+ 1668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:15:53.035761 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:15:53.037059 (+ 1298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:15:53.037066 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:15:53.037261 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:15:53.037326 (+ 65us) client_negotiation.cc:769] Sending connection context
0620 20:15:53.037468 (+ 142us) client_negotiation.cc:240] Negotiation successful
0620 20:15:53.037662 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3656,"mutex_wait_us":49,"thread_start_us":808,"threads_started":1}
I20260620 20:15:53.038928 19505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:53.026250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38800 (local address 127.16.217.1:40783)
0620 20:15:53.029062 (+ 2812us) server_negotiation.cc:206] Beginning negotiation
0620 20:15:53.029071 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 20:15:53.029104 (+ 33us) server_negotiation.cc:407] Connection header received
0620 20:15:53.029240 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:15:53.029251 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:15:53.029520 (+ 269us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:15:53.029706 (+ 186us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:15:53.030679 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:53.032561 (+ 1882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:53.035696 (+ 3135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:53.037650 (+ 1954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:53.038188 (+ 538us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:15:53.038391 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0620 20:15:53.038467 (+ 76us) server_negotiation.cc:299] Negotiation successful
0620 20:15:53.038636 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2658,"thread_start_us":2520,"threads_started":1}
I20260620 20:15:53.039189 19508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:53.028032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45486 (local address 127.16.217.2:40071)
0620 20:15:53.028834 (+ 802us) server_negotiation.cc:206] Beginning negotiation
0620 20:15:53.028847 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 20:15:53.031391 (+ 2544us) server_negotiation.cc:407] Connection header received
0620 20:15:53.031597 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:15:53.031602 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:15:53.031758 (+ 156us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:15:53.031944 (+ 186us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:15:53.032467 (+ 523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:53.033918 (+ 1451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:53.035908 (+ 1990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:53.036861 (+ 953us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:53.037026 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:15:53.037245 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0620 20:15:53.038681 (+ 1436us) server_negotiation.cc:299] Negotiation successful
0620 20:15:53.038873 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":565,"thread_start_us":390,"threads_started":1}
I20260620 20:15:53.039469 19506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:53.026721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41296 (local address 127.16.217.3:45785)
0620 20:15:53.029455 (+ 2734us) server_negotiation.cc:206] Beginning negotiation
0620 20:15:53.029462 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:15:53.029492 (+ 30us) server_negotiation.cc:407] Connection header received
0620 20:15:53.029582 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:15:53.029589 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:15:53.029786 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:15:53.029995 (+ 209us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:15:53.031444 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:53.033057 (+ 1613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:53.036010 (+ 2953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:53.037398 (+ 1388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:53.037517 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:15:53.037731 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0620 20:15:53.039000 (+ 1269us) server_negotiation.cc:299] Negotiation successful
0620 20:15:53.039145 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2572,"thread_start_us":2394,"threads_started":1}
I20260620 20:15:53.038862 19507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:53.026186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.3:45785 (local address 127.0.0.1:41296)
0620 20:15:53.028092 (+ 1906us) negotiation.cc:107] Waiting for socket to connect
0620 20:15:53.028121 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0620 20:15:53.028257 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:15:53.029985 (+ 1728us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:15:53.029990 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:15:53.030020 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:15:53.031136 (+ 1116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:15:53.031150 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:15:53.033196 (+ 2046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:15:53.033204 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:15:53.035887 (+ 2683us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:15:53.035900 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:15:53.037929 (+ 2029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:15:53.037937 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:15:53.038190 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:15:53.038254 (+ 64us) client_negotiation.cc:769] Sending connection context
0620 20:15:53.038340 (+ 86us) client_negotiation.cc:240] Negotiation successful
0620 20:15:53.038464 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1757,"mutex_wait_us":159,"thread_start_us":4255,"threads_started":1}
I20260620 20:15:53.038985 19503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:53.026064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.1:40783 (local address 127.0.0.1:38800)
0620 20:15:53.026931 (+ 867us) negotiation.cc:107] Waiting for socket to connect
0620 20:15:53.026979 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0620 20:15:53.027161 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:15:53.029759 (+ 2598us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:15:53.029772 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:15:53.029836 (+ 64us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:15:53.030462 (+ 626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:15:53.030476 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:15:53.032688 (+ 2212us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:15:53.032695 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:15:53.035533 (+ 2838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:15:53.035547 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:15:53.037946 (+ 2399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:15:53.037952 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:15:53.038256 (+ 304us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:15:53.038304 (+ 48us) client_negotiation.cc:769] Sending connection context
0620 20:15:53.038397 (+ 93us) client_negotiation.cc:240] Negotiation successful
0620 20:15:53.038536 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":564,"mutex_wait_us":223,"thread_start_us":434,"threads_started":1}
I20260620 20:15:53.043304 19281 tablet_service.cc:1467] Processing CreateTablet for tablet 677eb6aa3951473980bf32e063870661 (DEFAULT_TABLE table=test-table [id=059007b1134c416698015a3db0bf8dec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:15:53.043491 19420 tablet_service.cc:1467] Processing CreateTablet for tablet 677eb6aa3951473980bf32e063870661 (DEFAULT_TABLE table=test-table [id=059007b1134c416698015a3db0bf8dec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:15:53.044632 19420 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 677eb6aa3951473980bf32e063870661. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:15:53.044685 19281 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 677eb6aa3951473980bf32e063870661. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:15:53.044265 19141 tablet_service.cc:1467] Processing CreateTablet for tablet 677eb6aa3951473980bf32e063870661 (DEFAULT_TABLE table=test-table [id=059007b1134c416698015a3db0bf8dec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:15:53.045421 19141 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 677eb6aa3951473980bf32e063870661. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:15:53.055692 19509 tablet_bootstrap.cc:492] T 677eb6aa3951473980bf32e063870661 P 56cd29ec76f24959a145a593ab44b2d0: Bootstrap starting.
I20260620 20:15:53.058984 19510 tablet_bootstrap.cc:492] T 677eb6aa3951473980bf32e063870661 P ce1bf2b4620340b88840348ea13eb73d: Bootstrap starting.
I20260620 20:15:53.058980 19509 tablet_bootstrap.cc:654] T 677eb6aa3951473980bf32e063870661 P 56cd29ec76f24959a145a593ab44b2d0: Neither blocks nor log segments found. Creating new log.
I20260620 20:15:53.060663 19509 log.cc:824] T 677eb6aa3951473980bf32e063870661 P 56cd29ec76f24959a145a593ab44b2d0: Log is configured to *not* fsync() on all Append() calls
I20260620 20:15:53.060745 19511 tablet_bootstrap.cc:492] T 677eb6aa3951473980bf32e063870661 P 426541981bf84ca1a3e874674a81d687: Bootstrap starting.
I20260620 20:15:53.063341 19510 tablet_bootstrap.cc:654] T 677eb6aa3951473980bf32e063870661 P ce1bf2b4620340b88840348ea13eb73d: Neither blocks nor log segments found. Creating new log.
I20260620 20:15:53.063915 19511 tablet_bootstrap.cc:654] T 677eb6aa3951473980bf32e063870661 P 426541981bf84ca1a3e874674a81d687: Neither blocks nor log segments found. Creating new log.
I20260620 20:15:53.064065 19509 tablet_bootstrap.cc:492] T 677eb6aa3951473980bf32e063870661 P 56cd29ec76f24959a145a593ab44b2d0: No bootstrap required, opened a new log
I20260620 20:15:53.064342 19509 ts_tablet_manager.cc:1397] T 677eb6aa3951473980bf32e063870661 P 56cd29ec76f24959a145a593ab44b2d0: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.005s
I20260620 20:15:53.064571 19510 log.cc:824] T 677eb6aa3951473980bf32e063870661 P ce1bf2b4620340b88840348ea13eb73d: Log is configured to *not* fsync() on all Append() calls
I20260620 20:15:53.064987 19511 log.cc:824] T 677eb6aa3951473980bf32e063870661 P 426541981bf84ca1a3e874674a81d687: Log is configured to *not* fsync() on all Append() calls
I20260620 20:15:53.067404 19510 tablet_bootstrap.cc:492] T 677eb6aa3951473980bf32e063870661 P ce1bf2b4620340b88840348ea13eb73d: No bootstrap required, opened a new log
I20260620 20:15:53.067709 19510 ts_tablet_manager.cc:1397] T 677eb6aa3951473980bf32e063870661 P ce1bf2b4620340b88840348ea13eb73d: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260620 20:15:53.067916 19511 tablet_bootstrap.cc:492] T 677eb6aa3951473980bf32e063870661 P 426541981bf84ca1a3e874674a81d687: No bootstrap required, opened a new log
I20260620 20:15:53.068210 19511 ts_tablet_manager.cc:1397] T 677eb6aa3951473980bf32e063870661 P 426541981bf84ca1a3e874674a81d687: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260620 20:15:53.072234 19509 raft_consensus.cc:348] T 677eb6aa3951473980bf32e063870661 P 56cd29ec76f24959a145a593ab44b2d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "426541981bf84ca1a3e874674a81d687" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 40783 } } peers { permanent_uuid: "56cd29ec76f24959a145a593ab44b2d0" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 45785 } } peers { permanent_uuid: "ce1bf2b4620340b88840348ea13eb73d" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 40071 } }
I20260620 20:15:53.072599 19509 raft_consensus.cc:374] T 677eb6aa3951473980bf32e063870661 P 56cd29ec76f24959a145a593ab44b2d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:15:53.072712 19509 raft_consensus.cc:729] T 677eb6aa3951473980bf32e063870661 P 56cd29ec76f24959a145a593ab44b2d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56cd29ec76f24959a145a593ab44b2d0, State: Initialized, Role: FOLLOWER
I20260620 20:15:53.073328 19509 consensus_queue.cc:260] T 677eb6aa3951473980bf32e063870661 P 56cd29ec76f24959a145a593ab44b2d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "426541981bf84ca1a3e874674a81d687" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 40783 } } peers { permanent_uuid: "56cd29ec76f24959a145a593ab44b2d0" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 45785 } } peers { permanent_uuid: "ce1bf2b4620340b88840348ea13eb73d" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 40071 } }
I20260620 20:15:53.074420 19490 heartbeater.cc:503] Master 127.16.217.62:38007 was elected leader, sending a full tablet report...
I20260620 20:15:53.075063 19509 ts_tablet_manager.cc:1428] T 677eb6aa3951473980bf32e063870661 P 56cd29ec76f24959a145a593ab44b2d0: Time spent starting tablet: real 0.011s user 0.008s sys 0.004s
I20260620 20:15:53.078342 19510 raft_consensus.cc:348] T 677eb6aa3951473980bf32e063870661 P ce1bf2b4620340b88840348ea13eb73d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "426541981bf84ca1a3e874674a81d687" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 40783 } } peers { permanent_uuid: "56cd29ec76f24959a145a593ab44b2d0" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 45785 } } peers { permanent_uuid: "ce1bf2b4620340b88840348ea13eb73d" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 40071 } }
I20260620 20:15:53.078469 19511 raft_consensus.cc:348] T 677eb6aa3951473980bf32e063870661 P 426541981bf84ca1a3e874674a81d687 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "426541981bf84ca1a3e874674a81d687" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 40783 } } peers { permanent_uuid: "56cd29ec76f24959a145a593ab44b2d0" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 45785 } } peers { permanent_uuid: "ce1bf2b4620340b88840348ea13eb73d" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 40071 } }
I20260620 20:15:53.078846 19510 raft_consensus.cc:374] T 677eb6aa3951473980bf32e063870661 P ce1bf2b4620340b88840348ea13eb73d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:15:53.078857 19511 raft_consensus.cc:374] T 677eb6aa3951473980bf32e063870661 P 426541981bf84ca1a3e874674a81d687 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:15:53.078945 19511 raft_consensus.cc:729] T 677eb6aa3951473980bf32e063870661 P 426541981bf84ca1a3e874674a81d687 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 426541981bf84ca1a3e874674a81d687, State: Initialized, Role: FOLLOWER
I20260620 20:15:53.078994 19510 raft_consensus.cc:729] T 677eb6aa3951473980bf32e063870661 P ce1bf2b4620340b88840348ea13eb73d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce1bf2b4620340b88840348ea13eb73d, State: Initialized, Role: FOLLOWER
I20260620 20:15:53.079666 19511 consensus_queue.cc:260] T 677eb6aa3951473980bf32e063870661 P 426541981bf84ca1a3e874674a81d687 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "426541981bf84ca1a3e874674a81d687" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 40783 } } peers { permanent_uuid: "56cd29ec76f24959a145a593ab44b2d0" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 45785 } } peers { permanent_uuid: "ce1bf2b4620340b88840348ea13eb73d" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 40071 } }
I20260620 20:15:53.079669 19510 consensus_queue.cc:260] T 677eb6aa3951473980bf32e063870661 P ce1bf2b4620340b88840348ea13eb73d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "426541981bf84ca1a3e874674a81d687" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 40783 } } peers { permanent_uuid: "56cd29ec76f24959a145a593ab44b2d0" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 45785 } } peers { permanent_uuid: "ce1bf2b4620340b88840348ea13eb73d" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 40071 } }
I20260620 20:15:53.080718 19351 heartbeater.cc:503] Master 127.16.217.62:38007 was elected leader, sending a full tablet report...
I20260620 20:15:53.081403 19511 ts_tablet_manager.cc:1428] T 677eb6aa3951473980bf32e063870661 P 426541981bf84ca1a3e874674a81d687: Time spent starting tablet: real 0.013s user 0.012s sys 0.000s
I20260620 20:15:53.082476 19510 ts_tablet_manager.cc:1428] T 677eb6aa3951473980bf32e063870661 P ce1bf2b4620340b88840348ea13eb73d: Time spent starting tablet: real 0.015s user 0.009s sys 0.004s
W20260620 20:15:53.187515 19213 tablet.cc:2369] T 677eb6aa3951473980bf32e063870661 P 426541981bf84ca1a3e874674a81d687: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 20:15:53.191516 19352 tablet.cc:2369] T 677eb6aa3951473980bf32e063870661 P ce1bf2b4620340b88840348ea13eb73d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 20:15:53.197551 19491 tablet.cc:2369] T 677eb6aa3951473980bf32e063870661 P 56cd29ec76f24959a145a593ab44b2d0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 20:15:53.409579 19517 raft_consensus.cc:482] T 677eb6aa3951473980bf32e063870661 P 426541981bf84ca1a3e874674a81d687 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 20:15:53.409998 19517 raft_consensus.cc:504] T 677eb6aa3951473980bf32e063870661 P 426541981bf84ca1a3e874674a81d687 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "426541981bf84ca1a3e874674a81d687" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 40783 } } peers { permanent_uuid: "56cd29ec76f24959a145a593ab44b2d0" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 45785 } } peers { permanent_uuid: "ce1bf2b4620340b88840348ea13eb73d" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 40071 } }
I20260620 20:15:53.411983 19517 leader_election.cc:288] T 677eb6aa3951473980bf32e063870661 P 426541981bf84ca1a3e874674a81d687 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 56cd29ec76f24959a145a593ab44b2d0 (127.16.217.3:45785), ce1bf2b4620340b88840348ea13eb73d (127.16.217.2:40071)
I20260620 20:15:53.421638 19506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:53.412460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.217.1:48851 (local address 127.16.217.3:45785)
0620 20:15:53.412652 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0620 20:15:53.412659 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:15:53.413562 (+ 903us) server_negotiation.cc:407] Connection header received
0620 20:15:53.413755 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:15:53.413763 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:15:53.413844 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:15:53.414273 (+ 429us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:15:53.415033 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:53.416121 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:53.419177 (+ 3056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:53.420417 (+ 1240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:53.420586 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:15:53.420658 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 20:15:53.421228 (+ 570us) server_negotiation.cc:299] Negotiation successful
0620 20:15:53.421366 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 20:15:53.421646 19508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:53.412560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.217.1:47939 (local address 127.16.217.2:40071)
0620 20:15:53.412783 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0620 20:15:53.412791 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 20:15:53.413558 (+ 767us) server_negotiation.cc:407] Connection header received
0620 20:15:53.413973 (+ 415us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:15:53.413979 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:15:53.414044 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:15:53.414122 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:15:53.414986 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:53.416121 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:53.419188 (+ 3067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:53.420417 (+ 1229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:53.420586 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:15:53.420658 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 20:15:53.421228 (+ 570us) server_negotiation.cc:299] Negotiation successful
0620 20:15:53.421386 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260620 20:15:53.421952 19518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:53.412370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.3:45785 (local address 127.16.217.1:48851)
0620 20:15:53.413434 (+ 1064us) negotiation.cc:107] Waiting for socket to connect
0620 20:15:53.413452 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 20:15:53.413599 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:15:53.414333 (+ 734us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:15:53.414344 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:15:53.414381 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:15:53.414850 (+ 469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:15:53.414862 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:15:53.416270 (+ 1408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:15:53.416276 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:15:53.418944 (+ 2668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:15:53.418959 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:15:53.420719 (+ 1760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:15:53.420724 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:15:53.421005 (+ 281us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:15:53.421087 (+ 82us) client_negotiation.cc:769] Sending connection context
0620 20:15:53.421183 (+ 96us) client_negotiation.cc:240] Negotiation successful
0620 20:15:53.421365 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":942,"mutex_wait_us":284,"spinlock_wait_cycles":45568,"thread_start_us":518,"threads_started":1}
I20260620 20:15:53.421974 19519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:53.412278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.2:40071 (local address 127.16.217.1:47939)
0620 20:15:53.413397 (+ 1119us) negotiation.cc:107] Waiting for socket to connect
0620 20:15:53.413424 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0620 20:15:53.413594 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:15:53.414161 (+ 567us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:15:53.414165 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:15:53.414187 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:15:53.414753 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:15:53.414773 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:15:53.416270 (+ 1497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:15:53.416276 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:15:53.418951 (+ 2675us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:15:53.418959 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:15:53.420707 (+ 1748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:15:53.420714 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:15:53.421009 (+ 295us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:15:53.421085 (+ 76us) client_negotiation.cc:769] Sending connection context
0620 20:15:53.421183 (+ 98us) client_negotiation.cc:240] Negotiation successful
0620 20:15:53.421359 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":910,"mutex_wait_us":187,"spinlock_wait_cycles":27008,"thread_start_us":835,"threads_started":1}
I20260620 20:15:53.423192 19445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "677eb6aa3951473980bf32e063870661" candidate_uuid: "426541981bf84ca1a3e874674a81d687" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56cd29ec76f24959a145a593ab44b2d0" is_pre_election: true
I20260620 20:15:53.423192 19299 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "677eb6aa3951473980bf32e063870661" candidate_uuid: "426541981bf84ca1a3e874674a81d687" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce1bf2b4620340b88840348ea13eb73d" is_pre_election: true
I20260620 20:15:53.423718 19445 raft_consensus.cc:2454] T 677eb6aa3951473980bf32e063870661 P 56cd29ec76f24959a145a593ab44b2d0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 426541981bf84ca1a3e874674a81d687 in term 0.
I20260620 20:15:53.423718 19299 raft_consensus.cc:2454] T 677eb6aa3951473980bf32e063870661 P ce1bf2b4620340b88840348ea13eb73d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 426541981bf84ca1a3e874674a81d687 in term 0.
I20260620 20:15:53.424451 19091 leader_election.cc:302] T 677eb6aa3951473980bf32e063870661 P 426541981bf84ca1a3e874674a81d687 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 426541981bf84ca1a3e874674a81d687, 56cd29ec76f24959a145a593ab44b2d0, ce1bf2b4620340b88840348ea13eb73d; no voters:
I20260620 20:15:53.424911 19517 raft_consensus.cc:2781] T 677eb6aa3951473980bf32e063870661 P 426541981bf84ca1a3e874674a81d687 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 20:15:53.425068 19517 raft_consensus.cc:482] T 677eb6aa3951473980bf32e063870661 P 426541981bf84ca1a3e874674a81d687 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 20:15:53.425192 19517 raft_consensus.cc:3037] T 677eb6aa3951473980bf32e063870661 P 426541981bf84ca1a3e874674a81d687 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:15:53.427721 19517 raft_consensus.cc:504] T 677eb6aa3951473980bf32e063870661 P 426541981bf84ca1a3e874674a81d687 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "426541981bf84ca1a3e874674a81d687" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 40783 } } peers { permanent_uuid: "56cd29ec76f24959a145a593ab44b2d0" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 45785 } } peers { permanent_uuid: "ce1bf2b4620340b88840348ea13eb73d" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 40071 } }
I20260620 20:15:53.428493 19517 leader_election.cc:288] T 677eb6aa3951473980bf32e063870661 P 426541981bf84ca1a3e874674a81d687 [CANDIDATE]: Term 1 election: Requested vote from peers 56cd29ec76f24959a145a593ab44b2d0 (127.16.217.3:45785), ce1bf2b4620340b88840348ea13eb73d (127.16.217.2:40071)
I20260620 20:15:53.428965 19445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "677eb6aa3951473980bf32e063870661" candidate_uuid: "426541981bf84ca1a3e874674a81d687" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56cd29ec76f24959a145a593ab44b2d0"
I20260620 20:15:53.428989 19299 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "677eb6aa3951473980bf32e063870661" candidate_uuid: "426541981bf84ca1a3e874674a81d687" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce1bf2b4620340b88840348ea13eb73d"
I20260620 20:15:53.429170 19445 raft_consensus.cc:3037] T 677eb6aa3951473980bf32e063870661 P 56cd29ec76f24959a145a593ab44b2d0 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:15:53.429198 19299 raft_consensus.cc:3037] T 677eb6aa3951473980bf32e063870661 P ce1bf2b4620340b88840348ea13eb73d [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:15:53.431789 19445 raft_consensus.cc:2454] T 677eb6aa3951473980bf32e063870661 P 56cd29ec76f24959a145a593ab44b2d0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 426541981bf84ca1a3e874674a81d687 in term 1.
I20260620 20:15:53.432299 19094 leader_election.cc:302] T 677eb6aa3951473980bf32e063870661 P 426541981bf84ca1a3e874674a81d687 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 426541981bf84ca1a3e874674a81d687, 56cd29ec76f24959a145a593ab44b2d0; no voters:
I20260620 20:15:53.432484 19299 raft_consensus.cc:2454] T 677eb6aa3951473980bf32e063870661 P ce1bf2b4620340b88840348ea13eb73d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 426541981bf84ca1a3e874674a81d687 in term 1.
I20260620 20:15:53.432669 19517 raft_consensus.cc:2781] T 677eb6aa3951473980bf32e063870661 P 426541981bf84ca1a3e874674a81d687 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:15:53.433149 19517 raft_consensus.cc:686] T 677eb6aa3951473980bf32e063870661 P 426541981bf84ca1a3e874674a81d687 [term 1 LEADER]: Becoming Leader. State: Replica: 426541981bf84ca1a3e874674a81d687, State: Running, Role: LEADER
I20260620 20:15:53.433745 19517 consensus_queue.cc:237] T 677eb6aa3951473980bf32e063870661 P 426541981bf84ca1a3e874674a81d687 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "426541981bf84ca1a3e874674a81d687" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 40783 } } peers { permanent_uuid: "56cd29ec76f24959a145a593ab44b2d0" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 45785 } } peers { permanent_uuid: "ce1bf2b4620340b88840348ea13eb73d" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 40071 } }
I20260620 20:15:53.438647 19003 catalog_manager.cc:5495] T 677eb6aa3951473980bf32e063870661 P 426541981bf84ca1a3e874674a81d687 reported cstate change: term changed from 0 to 1, leader changed from <none> to 426541981bf84ca1a3e874674a81d687 (127.16.217.1). New cstate: current_term: 1 leader_uuid: "426541981bf84ca1a3e874674a81d687" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "426541981bf84ca1a3e874674a81d687" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 40783 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "56cd29ec76f24959a145a593ab44b2d0" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 45785 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce1bf2b4620340b88840348ea13eb73d" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 40071 } health_report { overall_health: UNKNOWN } } }
Jun 20 20:15:53 dist-test-slave-2hpg krb5kdc[18959](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986551, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.217.62@KRBTEST.COM for kudu/127.16.217.62@KRBTEST.COM
I20260620 20:15:53.529913 19528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:53.516133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:38007 (local address 127.0.0.1:55280)
0620 20:15:53.517010 (+ 877us) negotiation.cc:107] Waiting for socket to connect
0620 20:15:53.517029 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 20:15:53.517204 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:15:53.517541 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:15:53.517546 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:15:53.518256 (+ 710us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:15:53.518490 (+ 234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:15:53.518499 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:15:53.519560 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:15:53.519563 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:15:53.520480 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:15:53.520489 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:15:53.521153 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:15:53.521157 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:15:53.521273 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:15:53.521929 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:15:53.521948 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:15:53.524102 (+ 2154us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:15:53.527245 (+ 3143us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:15:53.527259 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:15:53.527268 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:15:53.527705 (+ 437us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:15:53.528081 (+ 376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:15:53.528086 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:15:53.528090 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:15:53.528229 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:15:53.528763 (+ 534us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:15:53.528772 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:15:53.528948 (+ 176us) client_negotiation.cc:769] Sending connection context
0620 20:15:53.529122 (+ 174us) client_negotiation.cc:240] Negotiation successful
0620 20:15:53.529404 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":739,"spinlock_wait_cycles":16256,"thread_start_us":480,"threads_started":1}
I20260620 20:15:53.529999 19527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:53.516133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55280 (local address 127.16.217.62:38007)
0620 20:15:53.516942 (+ 809us) server_negotiation.cc:206] Beginning negotiation
0620 20:15:53.516948 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:15:53.517117 (+ 169us) server_negotiation.cc:407] Connection header received
0620 20:15:53.517315 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:15:53.517322 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:15:53.517390 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:15:53.517498 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:15:53.518644 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:53.519452 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:53.520595 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:53.520998 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:53.521088 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:15:53.524331 (+ 3243us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:15:53.524336 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:15:53.524339 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:15:53.524366 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:15:53.527094 (+ 2728us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:15:53.527843 (+ 749us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:15:53.527849 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:15:53.527853 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:15:53.527966 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:15:53.528349 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:15:53.528356 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:15:53.528360 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:15:53.528627 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:15:53.528774 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0620 20:15:53.529209 (+ 435us) server_negotiation.cc:299] Negotiation successful
0620 20:15:53.529542 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":133,"server-negotiator.queue_time_us":636,"thread_start_us":461,"threads_started":1}
I20260620 20:15:53.535141 19003 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.16.217.62@KRBTEST.COM'} at 127.0.0.1:55280:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260620 20:15:53.536299 19003 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 20:15:53.542982 19141 tablet_service.cc:1467] Processing CreateTablet for tablet 41ded77a1c204a8ebaa1c3f08019c7c4 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=b756ab9ec22e491b8a8c1fad0bf6cdaf]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 20:15:53.543116 19281 tablet_service.cc:1467] Processing CreateTablet for tablet 41ded77a1c204a8ebaa1c3f08019c7c4 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=b756ab9ec22e491b8a8c1fad0bf6cdaf]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 20:15:53.543888 19141 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 41ded77a1c204a8ebaa1c3f08019c7c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:15:53.543498 19420 tablet_service.cc:1467] Processing CreateTablet for tablet 41ded77a1c204a8ebaa1c3f08019c7c4 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=b756ab9ec22e491b8a8c1fad0bf6cdaf]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 20:15:53.543967 19281 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 41ded77a1c204a8ebaa1c3f08019c7c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:15:53.544198 19420 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 41ded77a1c204a8ebaa1c3f08019c7c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:15:53.549964 19511 tablet_bootstrap.cc:492] T 41ded77a1c204a8ebaa1c3f08019c7c4 P 426541981bf84ca1a3e874674a81d687: Bootstrap starting.
I20260620 20:15:53.551050 19509 tablet_bootstrap.cc:492] T 41ded77a1c204a8ebaa1c3f08019c7c4 P 56cd29ec76f24959a145a593ab44b2d0: Bootstrap starting.
I20260620 20:15:53.551446 19511 tablet_bootstrap.cc:654] T 41ded77a1c204a8ebaa1c3f08019c7c4 P 426541981bf84ca1a3e874674a81d687: Neither blocks nor log segments found. Creating new log.
I20260620 20:15:53.552742 19509 tablet_bootstrap.cc:654] T 41ded77a1c204a8ebaa1c3f08019c7c4 P 56cd29ec76f24959a145a593ab44b2d0: Neither blocks nor log segments found. Creating new log.
I20260620 20:15:53.553534 19511 tablet_bootstrap.cc:492] T 41ded77a1c204a8ebaa1c3f08019c7c4 P 426541981bf84ca1a3e874674a81d687: No bootstrap required, opened a new log
I20260620 20:15:53.553659 19511 ts_tablet_manager.cc:1397] T 41ded77a1c204a8ebaa1c3f08019c7c4 P 426541981bf84ca1a3e874674a81d687: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260620 20:15:53.553707 19510 tablet_bootstrap.cc:492] T 41ded77a1c204a8ebaa1c3f08019c7c4 P ce1bf2b4620340b88840348ea13eb73d: Bootstrap starting.
I20260620 20:15:53.554443 19511 raft_consensus.cc:348] T 41ded77a1c204a8ebaa1c3f08019c7c4 P 426541981bf84ca1a3e874674a81d687 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "426541981bf84ca1a3e874674a81d687" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 40783 } } peers { permanent_uuid: "ce1bf2b4620340b88840348ea13eb73d" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 40071 } } peers { permanent_uuid: "56cd29ec76f24959a145a593ab44b2d0" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 45785 } }
I20260620 20:15:53.554674 19511 raft_consensus.cc:374] T 41ded77a1c204a8ebaa1c3f08019c7c4 P 426541981bf84ca1a3e874674a81d687 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:15:53.554754 19511 raft_consensus.cc:729] T 41ded77a1c204a8ebaa1c3f08019c7c4 P 426541981bf84ca1a3e874674a81d687 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 426541981bf84ca1a3e874674a81d687, State: Initialized, Role: FOLLOWER
I20260620 20:15:53.554972 19511 consensus_queue.cc:260] T 41ded77a1c204a8ebaa1c3f08019c7c4 P 426541981bf84ca1a3e874674a81d687 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "426541981bf84ca1a3e874674a81d687" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 40783 } } peers { permanent_uuid: "ce1bf2b4620340b88840348ea13eb73d" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 40071 } } peers { permanent_uuid: "56cd29ec76f24959a145a593ab44b2d0" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 45785 } }
I20260620 20:15:53.555297 19510 tablet_bootstrap.cc:654] T 41ded77a1c204a8ebaa1c3f08019c7c4 P ce1bf2b4620340b88840348ea13eb73d: Neither blocks nor log segments found. Creating new log.
I20260620 20:15:53.555414 19517 tablet_replica.cc:439] T 41ded77a1c204a8ebaa1c3f08019c7c4 P 426541981bf84ca1a3e874674a81d687: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "426541981bf84ca1a3e874674a81d687" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 40783 } } peers { permanent_uuid: "ce1bf2b4620340b88840348ea13eb73d" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 40071 } } peers { permanent_uuid: "56cd29ec76f24959a145a593ab44b2d0" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 45785 } } }
I20260620 20:15:53.555655 19511 ts_tablet_manager.cc:1428] T 41ded77a1c204a8ebaa1c3f08019c7c4 P 426541981bf84ca1a3e874674a81d687: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260620 20:15:53.555711 19517 tablet_replica.cc:442] T 41ded77a1c204a8ebaa1c3f08019c7c4 P 426541981bf84ca1a3e874674a81d687: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 20:15:53.557468 19509 tablet_bootstrap.cc:492] T 41ded77a1c204a8ebaa1c3f08019c7c4 P 56cd29ec76f24959a145a593ab44b2d0: No bootstrap required, opened a new log
I20260620 20:15:53.557608 19509 ts_tablet_manager.cc:1397] T 41ded77a1c204a8ebaa1c3f08019c7c4 P 56cd29ec76f24959a145a593ab44b2d0: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.001s
I20260620 20:15:53.558357 19509 raft_consensus.cc:348] T 41ded77a1c204a8ebaa1c3f08019c7c4 P 56cd29ec76f24959a145a593ab44b2d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "426541981bf84ca1a3e874674a81d687" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 40783 } } peers { permanent_uuid: "ce1bf2b4620340b88840348ea13eb73d" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 40071 } } peers { permanent_uuid: "56cd29ec76f24959a145a593ab44b2d0" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 45785 } }
I20260620 20:15:53.558560 19509 raft_consensus.cc:374] T 41ded77a1c204a8ebaa1c3f08019c7c4 P 56cd29ec76f24959a145a593ab44b2d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:15:53.558651 19509 raft_consensus.cc:729] T 41ded77a1c204a8ebaa1c3f08019c7c4 P 56cd29ec76f24959a145a593ab44b2d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56cd29ec76f24959a145a593ab44b2d0, State: Initialized, Role: FOLLOWER
I20260620 20:15:53.558868 19509 consensus_queue.cc:260] T 41ded77a1c204a8ebaa1c3f08019c7c4 P 56cd29ec76f24959a145a593ab44b2d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "426541981bf84ca1a3e874674a81d687" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 40783 } } peers { permanent_uuid: "ce1bf2b4620340b88840348ea13eb73d" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 40071 } } peers { permanent_uuid: "56cd29ec76f24959a145a593ab44b2d0" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 45785 } }
I20260620 20:15:53.559652 19510 tablet_bootstrap.cc:492] T 41ded77a1c204a8ebaa1c3f08019c7c4 P ce1bf2b4620340b88840348ea13eb73d: No bootstrap required, opened a new log
I20260620 20:15:53.559831 19510 ts_tablet_manager.cc:1397] T 41ded77a1c204a8ebaa1c3f08019c7c4 P ce1bf2b4620340b88840348ea13eb73d: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20260620 20:15:53.560464 19510 raft_consensus.cc:348] T 41ded77a1c204a8ebaa1c3f08019c7c4 P ce1bf2b4620340b88840348ea13eb73d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "426541981bf84ca1a3e874674a81d687" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 40783 } } peers { permanent_uuid: "ce1bf2b4620340b88840348ea13eb73d" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 40071 } } peers { permanent_uuid: "56cd29ec76f24959a145a593ab44b2d0" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 45785 } }
I20260620 20:15:53.560669 19510 raft_consensus.cc:374] T 41ded77a1c204a8ebaa1c3f08019c7c4 P ce1bf2b4620340b88840348ea13eb73d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:15:53.560786 19510 raft_consensus.cc:729] T 41ded77a1c204a8ebaa1c3f08019c7c4 P ce1bf2b4620340b88840348ea13eb73d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce1bf2b4620340b88840348ea13eb73d, State: Initialized, Role: FOLLOWER
I20260620 20:15:53.561030 19510 consensus_queue.cc:260] T 41ded77a1c204a8ebaa1c3f08019c7c4 P ce1bf2b4620340b88840348ea13eb73d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "426541981bf84ca1a3e874674a81d687" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 40783 } } peers { permanent_uuid: "ce1bf2b4620340b88840348ea13eb73d" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 40071 } } peers { permanent_uuid: "56cd29ec76f24959a145a593ab44b2d0" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 45785 } }
I20260620 20:15:53.561692 19510 ts_tablet_manager.cc:1428] T 41ded77a1c204a8ebaa1c3f08019c7c4 P ce1bf2b4620340b88840348ea13eb73d: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260620 20:15:53.561468 19516 tablet_replica.cc:439] T 41ded77a1c204a8ebaa1c3f08019c7c4 P ce1bf2b4620340b88840348ea13eb73d: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "426541981bf84ca1a3e874674a81d687" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 40783 } } peers { permanent_uuid: "ce1bf2b4620340b88840348ea13eb73d" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 40071 } } peers { permanent_uuid: "56cd29ec76f24959a145a593ab44b2d0" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 45785 } } }
I20260620 20:15:53.561861 19516 tablet_replica.cc:442] T 41ded77a1c204a8ebaa1c3f08019c7c4 P ce1bf2b4620340b88840348ea13eb73d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 20:15:53.562119 19515 tablet_replica.cc:439] T 41ded77a1c204a8ebaa1c3f08019c7c4 P 56cd29ec76f24959a145a593ab44b2d0: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "426541981bf84ca1a3e874674a81d687" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 40783 } } peers { permanent_uuid: "ce1bf2b4620340b88840348ea13eb73d" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 40071 } } peers { permanent_uuid: "56cd29ec76f24959a145a593ab44b2d0" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 45785 } } }
I20260620 20:15:53.562465 19515 tablet_replica.cc:442] T 41ded77a1c204a8ebaa1c3f08019c7c4 P 56cd29ec76f24959a145a593ab44b2d0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 20:15:53.562294 19509 ts_tablet_manager.cc:1428] T 41ded77a1c204a8ebaa1c3f08019c7c4 P 56cd29ec76f24959a145a593ab44b2d0: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20260620 20:15:53.646353 19517 raft_consensus.cc:482] T 41ded77a1c204a8ebaa1c3f08019c7c4 P 426541981bf84ca1a3e874674a81d687 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 20:15:53.646556 19517 raft_consensus.cc:504] T 41ded77a1c204a8ebaa1c3f08019c7c4 P 426541981bf84ca1a3e874674a81d687 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "426541981bf84ca1a3e874674a81d687" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 40783 } } peers { permanent_uuid: "ce1bf2b4620340b88840348ea13eb73d" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 40071 } } peers { permanent_uuid: "56cd29ec76f24959a145a593ab44b2d0" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 45785 } }
I20260620 20:15:53.647162 19517 leader_election.cc:288] T 41ded77a1c204a8ebaa1c3f08019c7c4 P 426541981bf84ca1a3e874674a81d687 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ce1bf2b4620340b88840348ea13eb73d (127.16.217.2:40071), 56cd29ec76f24959a145a593ab44b2d0 (127.16.217.3:45785)
I20260620 20:15:53.647737 19299 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41ded77a1c204a8ebaa1c3f08019c7c4" candidate_uuid: "426541981bf84ca1a3e874674a81d687" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce1bf2b4620340b88840348ea13eb73d" is_pre_election: true
I20260620 20:15:53.647763 19445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41ded77a1c204a8ebaa1c3f08019c7c4" candidate_uuid: "426541981bf84ca1a3e874674a81d687" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56cd29ec76f24959a145a593ab44b2d0" is_pre_election: true
I20260620 20:15:53.647990 19299 raft_consensus.cc:2454] T 41ded77a1c204a8ebaa1c3f08019c7c4 P ce1bf2b4620340b88840348ea13eb73d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 426541981bf84ca1a3e874674a81d687 in term 0.
I20260620 20:15:53.648037 19445 raft_consensus.cc:2454] T 41ded77a1c204a8ebaa1c3f08019c7c4 P 56cd29ec76f24959a145a593ab44b2d0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 426541981bf84ca1a3e874674a81d687 in term 0.
I20260620 20:15:53.648437 19091 leader_election.cc:302] T 41ded77a1c204a8ebaa1c3f08019c7c4 P 426541981bf84ca1a3e874674a81d687 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 426541981bf84ca1a3e874674a81d687, ce1bf2b4620340b88840348ea13eb73d; no voters:
I20260620 20:15:53.648764 19517 raft_consensus.cc:2781] T 41ded77a1c204a8ebaa1c3f08019c7c4 P 426541981bf84ca1a3e874674a81d687 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 20:15:53.648871 19517 raft_consensus.cc:482] T 41ded77a1c204a8ebaa1c3f08019c7c4 P 426541981bf84ca1a3e874674a81d687 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 20:15:53.648957 19517 raft_consensus.cc:3037] T 41ded77a1c204a8ebaa1c3f08019c7c4 P 426541981bf84ca1a3e874674a81d687 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:15:53.650843 19517 raft_consensus.cc:504] T 41ded77a1c204a8ebaa1c3f08019c7c4 P 426541981bf84ca1a3e874674a81d687 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "426541981bf84ca1a3e874674a81d687" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 40783 } } peers { permanent_uuid: "ce1bf2b4620340b88840348ea13eb73d" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 40071 } } peers { permanent_uuid: "56cd29ec76f24959a145a593ab44b2d0" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 45785 } }
I20260620 20:15:53.651388 19517 leader_election.cc:288] T 41ded77a1c204a8ebaa1c3f08019c7c4 P 426541981bf84ca1a3e874674a81d687 [CANDIDATE]: Term 1 election: Requested vote from peers ce1bf2b4620340b88840348ea13eb73d (127.16.217.2:40071), 56cd29ec76f24959a145a593ab44b2d0 (127.16.217.3:45785)
I20260620 20:15:53.651763 19299 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41ded77a1c204a8ebaa1c3f08019c7c4" candidate_uuid: "426541981bf84ca1a3e874674a81d687" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce1bf2b4620340b88840348ea13eb73d"
I20260620 20:15:53.651858 19445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41ded77a1c204a8ebaa1c3f08019c7c4" candidate_uuid: "426541981bf84ca1a3e874674a81d687" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56cd29ec76f24959a145a593ab44b2d0"
I20260620 20:15:53.651986 19299 raft_consensus.cc:3037] T 41ded77a1c204a8ebaa1c3f08019c7c4 P ce1bf2b4620340b88840348ea13eb73d [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:15:53.652064 19445 raft_consensus.cc:3037] T 41ded77a1c204a8ebaa1c3f08019c7c4 P 56cd29ec76f24959a145a593ab44b2d0 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:15:53.654825 19445 raft_consensus.cc:2454] T 41ded77a1c204a8ebaa1c3f08019c7c4 P 56cd29ec76f24959a145a593ab44b2d0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 426541981bf84ca1a3e874674a81d687 in term 1.
I20260620 20:15:53.654825 19299 raft_consensus.cc:2454] T 41ded77a1c204a8ebaa1c3f08019c7c4 P ce1bf2b4620340b88840348ea13eb73d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 426541981bf84ca1a3e874674a81d687 in term 1.
I20260620 20:15:53.655195 19094 leader_election.cc:302] T 41ded77a1c204a8ebaa1c3f08019c7c4 P 426541981bf84ca1a3e874674a81d687 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 426541981bf84ca1a3e874674a81d687, 56cd29ec76f24959a145a593ab44b2d0; no voters:
I20260620 20:15:53.655478 19517 raft_consensus.cc:2781] T 41ded77a1c204a8ebaa1c3f08019c7c4 P 426541981bf84ca1a3e874674a81d687 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:15:53.655655 19517 raft_consensus.cc:686] T 41ded77a1c204a8ebaa1c3f08019c7c4 P 426541981bf84ca1a3e874674a81d687 [term 1 LEADER]: Becoming Leader. State: Replica: 426541981bf84ca1a3e874674a81d687, State: Running, Role: LEADER
I20260620 20:15:53.655910 19517 consensus_queue.cc:237] T 41ded77a1c204a8ebaa1c3f08019c7c4 P 426541981bf84ca1a3e874674a81d687 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "426541981bf84ca1a3e874674a81d687" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 40783 } } peers { permanent_uuid: "ce1bf2b4620340b88840348ea13eb73d" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 40071 } } peers { permanent_uuid: "56cd29ec76f24959a145a593ab44b2d0" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 45785 } }
I20260620 20:15:53.657449 19520 tablet_replica.cc:439] T 41ded77a1c204a8ebaa1c3f08019c7c4 P 426541981bf84ca1a3e874674a81d687: TxnStatusTablet state changed. Reason: New leader 426541981bf84ca1a3e874674a81d687. Latest consensus state: current_term: 1 leader_uuid: "426541981bf84ca1a3e874674a81d687" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "426541981bf84ca1a3e874674a81d687" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 40783 } } peers { permanent_uuid: "ce1bf2b4620340b88840348ea13eb73d" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 40071 } } peers { permanent_uuid: "56cd29ec76f24959a145a593ab44b2d0" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 45785 } } }
I20260620 20:15:53.657717 19520 tablet_replica.cc:442] T 41ded77a1c204a8ebaa1c3f08019c7c4 P 426541981bf84ca1a3e874674a81d687: This TxnStatusTablet replica's current role is: LEADER
I20260620 20:15:53.658385 19533 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260620 20:15:53.658661 19533 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260620 20:15:53.658833 19004 catalog_manager.cc:5495] T 41ded77a1c204a8ebaa1c3f08019c7c4 P 426541981bf84ca1a3e874674a81d687 reported cstate change: term changed from 0 to 1, leader changed from <none> to 426541981bf84ca1a3e874674a81d687 (127.16.217.1). New cstate: current_term: 1 leader_uuid: "426541981bf84ca1a3e874674a81d687" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "426541981bf84ca1a3e874674a81d687" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 40783 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce1bf2b4620340b88840348ea13eb73d" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 40071 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "56cd29ec76f24959a145a593ab44b2d0" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 45785 } health_report { overall_health: UNKNOWN } } }
I20260620 20:15:53.679895 19536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:53.673072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38812 (local address 127.16.217.1:40783)
0620 20:15:53.673724 (+ 652us) server_negotiation.cc:206] Beginning negotiation
0620 20:15:53.673729 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:15:53.673748 (+ 19us) server_negotiation.cc:407] Connection header received
0620 20:15:53.673829 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:15:53.673832 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:15:53.673968 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:15:53.674110 (+ 142us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 20:15:53.674553 (+ 443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:53.675505 (+ 952us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:53.677378 (+ 1873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:53.677943 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:53.678044 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:15:53.678591 (+ 547us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 20:15:53.679186 (+ 595us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 20:15:53.679314 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0620 20:15:53.679456 (+ 142us) server_negotiation.cc:299] Negotiation successful
0620 20:15:53.679522 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":527,"thread_start_us":378,"threads_started":1}
I20260620 20:15:53.679888 19528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:53.672918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.1:40783 (local address 127.0.0.1:38812)
0620 20:15:53.673188 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0620 20:15:53.673209 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 20:15:53.673343 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:15:53.674163 (+ 820us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:15:53.674172 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:15:53.674188 (+ 16us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 20:15:53.674399 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:15:53.674410 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:15:53.675636 (+ 1226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:15:53.675642 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:15:53.677227 (+ 1585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:15:53.677241 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:15:53.678084 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:15:53.678091 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:15:53.678257 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:15:53.678323 (+ 66us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 20:15:53.679307 (+ 984us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 20:15:53.679326 (+ 19us) client_negotiation.cc:769] Sending connection context
0620 20:15:53.679425 (+ 99us) client_negotiation.cc:240] Negotiation successful
0620 20:15:53.679577 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":84}
I20260620 20:15:53.687929 19445 raft_consensus.cc:1261] T 41ded77a1c204a8ebaa1c3f08019c7c4 P 56cd29ec76f24959a145a593ab44b2d0 [term 1 FOLLOWER]: Refusing update from remote peer 426541981bf84ca1a3e874674a81d687: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:15:53.688107 19299 raft_consensus.cc:1261] T 41ded77a1c204a8ebaa1c3f08019c7c4 P ce1bf2b4620340b88840348ea13eb73d [term 1 FOLLOWER]: Refusing update from remote peer 426541981bf84ca1a3e874674a81d687: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:15:53.688896 19520 consensus_queue.cc:1035] T 41ded77a1c204a8ebaa1c3f08019c7c4 P 426541981bf84ca1a3e874674a81d687 [LEADER]: Connected to new peer: Peer: permanent_uuid: "56cd29ec76f24959a145a593ab44b2d0" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 45785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:15:53.689240 19517 consensus_queue.cc:1035] T 41ded77a1c204a8ebaa1c3f08019c7c4 P 426541981bf84ca1a3e874674a81d687 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce1bf2b4620340b88840348ea13eb73d" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 40071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:15:53.697579 19516 tablet_replica.cc:439] T 41ded77a1c204a8ebaa1c3f08019c7c4 P ce1bf2b4620340b88840348ea13eb73d: TxnStatusTablet state changed. Reason: New leader 426541981bf84ca1a3e874674a81d687. Latest consensus state: current_term: 1 leader_uuid: "426541981bf84ca1a3e874674a81d687" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "426541981bf84ca1a3e874674a81d687" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 40783 } } peers { permanent_uuid: "ce1bf2b4620340b88840348ea13eb73d" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 40071 } } peers { permanent_uuid: "56cd29ec76f24959a145a593ab44b2d0" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 45785 } } }
I20260620 20:15:53.697932 19516 tablet_replica.cc:442] T 41ded77a1c204a8ebaa1c3f08019c7c4 P ce1bf2b4620340b88840348ea13eb73d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 20:15:53.700193 19537 mvcc.cc:205] Tried to move back new op lower bound from 7299016923895332864 to 7299016923776671744. Current Snapshot: MvccSnapshot[applied={T|T < 7299016923895332864}]
I20260620 20:15:53.702296 19517 tablet_replica.cc:439] T 41ded77a1c204a8ebaa1c3f08019c7c4 P 426541981bf84ca1a3e874674a81d687: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "426541981bf84ca1a3e874674a81d687" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "426541981bf84ca1a3e874674a81d687" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 40783 } } peers { permanent_uuid: "ce1bf2b4620340b88840348ea13eb73d" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 40071 } } peers { permanent_uuid: "56cd29ec76f24959a145a593ab44b2d0" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 45785 } } }
I20260620 20:15:53.702608 19517 tablet_replica.cc:442] T 41ded77a1c204a8ebaa1c3f08019c7c4 P 426541981bf84ca1a3e874674a81d687: This TxnStatusTablet replica's current role is: LEADER
I20260620 20:15:53.703889 19516 tablet_replica.cc:439] T 41ded77a1c204a8ebaa1c3f08019c7c4 P ce1bf2b4620340b88840348ea13eb73d: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "426541981bf84ca1a3e874674a81d687" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "426541981bf84ca1a3e874674a81d687" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 40783 } } peers { permanent_uuid: "ce1bf2b4620340b88840348ea13eb73d" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 40071 } } peers { permanent_uuid: "56cd29ec76f24959a145a593ab44b2d0" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 45785 } } }
I20260620 20:15:53.704100 19516 tablet_replica.cc:442] T 41ded77a1c204a8ebaa1c3f08019c7c4 P ce1bf2b4620340b88840348ea13eb73d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 20:15:53.706769 19515 tablet_replica.cc:439] T 41ded77a1c204a8ebaa1c3f08019c7c4 P 56cd29ec76f24959a145a593ab44b2d0: TxnStatusTablet state changed. Reason: New leader 426541981bf84ca1a3e874674a81d687. Latest consensus state: current_term: 1 leader_uuid: "426541981bf84ca1a3e874674a81d687" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "426541981bf84ca1a3e874674a81d687" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 40783 } } peers { permanent_uuid: "ce1bf2b4620340b88840348ea13eb73d" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 40071 } } peers { permanent_uuid: "56cd29ec76f24959a145a593ab44b2d0" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 45785 } } }
I20260620 20:15:53.707029 19515 tablet_replica.cc:442] T 41ded77a1c204a8ebaa1c3f08019c7c4 P 56cd29ec76f24959a145a593ab44b2d0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 20:15:53.710420 19517 tablet_replica.cc:439] T 41ded77a1c204a8ebaa1c3f08019c7c4 P 426541981bf84ca1a3e874674a81d687: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "426541981bf84ca1a3e874674a81d687" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "426541981bf84ca1a3e874674a81d687" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 40783 } } peers { permanent_uuid: "ce1bf2b4620340b88840348ea13eb73d" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 40071 } } peers { permanent_uuid: "56cd29ec76f24959a145a593ab44b2d0" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 45785 } } }
I20260620 20:15:53.710661 19517 tablet_replica.cc:442] T 41ded77a1c204a8ebaa1c3f08019c7c4 P 426541981bf84ca1a3e874674a81d687: This TxnStatusTablet replica's current role is: LEADER
I20260620 20:15:53.713650 19515 tablet_replica.cc:439] T 41ded77a1c204a8ebaa1c3f08019c7c4 P 56cd29ec76f24959a145a593ab44b2d0: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "426541981bf84ca1a3e874674a81d687" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "426541981bf84ca1a3e874674a81d687" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 40783 } } peers { permanent_uuid: "ce1bf2b4620340b88840348ea13eb73d" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 40071 } } peers { permanent_uuid: "56cd29ec76f24959a145a593ab44b2d0" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 45785 } } }
I20260620 20:15:53.713902 19515 tablet_replica.cc:442] T 41ded77a1c204a8ebaa1c3f08019c7c4 P 56cd29ec76f24959a145a593ab44b2d0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 20:15:53.719403 19536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:53.712269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38824 (local address 127.16.217.1:40783)
0620 20:15:53.712436 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0620 20:15:53.712443 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:15:53.713240 (+ 797us) server_negotiation.cc:407] Connection header received
0620 20:15:53.713604 (+ 364us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:15:53.713612 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:15:53.713699 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:15:53.713760 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 20:15:53.714473 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:53.715597 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:53.717659 (+ 2062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:53.718172 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:53.718313 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:15:53.718697 (+ 384us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 20:15:53.718794 (+ 97us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 20:15:53.718896 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0620 20:15:53.719033 (+ 137us) server_negotiation.cc:299] Negotiation successful
0620 20:15:53.719112 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 20:15:53.736953 19549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:53.729253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.1:40783 (local address 127.16.217.1:54311)
0620 20:15:53.729947 (+ 694us) negotiation.cc:107] Waiting for socket to connect
0620 20:15:53.729973 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0620 20:15:53.730128 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:15:53.730477 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:15:53.730482 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:15:53.730496 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 20:15:53.730994 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:15:53.731013 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:15:53.732501 (+ 1488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:15:53.732507 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:15:53.734549 (+ 2042us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:15:53.734566 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:15:53.735349 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:15:53.735356 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:15:53.735583 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:15:53.735654 (+ 71us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 20:15:53.736279 (+ 625us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 20:15:53.736290 (+ 11us) client_negotiation.cc:769] Sending connection context
0620 20:15:53.736385 (+ 95us) client_negotiation.cc:240] Negotiation successful
0620 20:15:53.736520 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":498,"thread_start_us":317,"threads_started":1}
I20260620 20:15:53.737004 19536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:53.729574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.217.1:54311 (local address 127.16.217.1:40783)
0620 20:15:53.729752 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0620 20:15:53.729760 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 20:15:53.730053 (+ 293us) server_negotiation.cc:407] Connection header received
0620 20:15:53.730259 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:15:53.730263 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:15:53.730327 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:15:53.730421 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 20:15:53.731144 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:53.732374 (+ 1230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:53.734705 (+ 2331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:53.735157 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:53.735253 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:15:53.735939 (+ 686us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 20:15:53.736102 (+ 163us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 20:15:53.736216 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0620 20:15:53.736414 (+ 198us) server_negotiation.cc:299] Negotiation successful
0620 20:15:53.736508 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":29,"server-negotiator.queue_time_us":68}
I20260620 20:15:53.747524 19299 raft_consensus.cc:1261] T 677eb6aa3951473980bf32e063870661 P ce1bf2b4620340b88840348ea13eb73d [term 1 FOLLOWER]: Refusing update from remote peer 426541981bf84ca1a3e874674a81d687: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:15:53.747717 19445 raft_consensus.cc:1261] T 677eb6aa3951473980bf32e063870661 P 56cd29ec76f24959a145a593ab44b2d0 [term 1 FOLLOWER]: Refusing update from remote peer 426541981bf84ca1a3e874674a81d687: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:15:53.748001 19550 consensus_queue.cc:1035] T 677eb6aa3951473980bf32e063870661 P 426541981bf84ca1a3e874674a81d687 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce1bf2b4620340b88840348ea13eb73d" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 40071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:15:53.748358 19520 consensus_queue.cc:1035] T 677eb6aa3951473980bf32e063870661 P 426541981bf84ca1a3e874674a81d687 [LEADER]: Connected to new peer: Peer: permanent_uuid: "56cd29ec76f24959a145a593ab44b2d0" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 45785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:15:53.758953 19539 mvcc.cc:205] Tried to move back new op lower bound from 7299016924143669248 to 7299016922869129216. Current Snapshot: MvccSnapshot[applied={T|T < 7299016924143669248}]
I20260620 20:15:53.812408 19004 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55264:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 20:15:53.812685 19004 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55264:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 20:15:53.816740 19004 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ebfe81c8d02342e6bc65b45342cd2b9d: Sending DeleteTablet for 3 replicas of tablet 677eb6aa3951473980bf32e063870661
I20260620 20:15:53.817900 19420 tablet_service.cc:1514] Processing DeleteTablet for tablet 677eb6aa3951473980bf32e063870661 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:15:53 UTC) from {username='kudu', principal='kudu/127.16.217.62@KRBTEST.COM'} at 127.0.0.1:41296
I20260620 20:15:53.817974 19281 tablet_service.cc:1514] Processing DeleteTablet for tablet 677eb6aa3951473980bf32e063870661 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:15:53 UTC) from {username='kudu', principal='kudu/127.16.217.62@KRBTEST.COM'} at 127.0.0.1:45486
I20260620 20:15:53.818114 19140 tablet_service.cc:1514] Processing DeleteTablet for tablet 677eb6aa3951473980bf32e063870661 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:15:53 UTC) from {username='kudu', principal='kudu/127.16.217.62@KRBTEST.COM'} at 127.0.0.1:38800
I20260620 20:15:53.820178 19004 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55264:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 20:15:53.821017 19004 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 20:15:53.828773 19280 tablet_service.cc:1467] Processing CreateTablet for tablet b90037163b8444b0a63eadde5fc42cd4 (DEFAULT_TABLE table=test-table [id=42b36ce6129d456697af0b9e0c994082]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:15:53.829329 19280 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b90037163b8444b0a63eadde5fc42cd4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:15:53.829602 19141 tablet_service.cc:1467] Processing CreateTablet for tablet b90037163b8444b0a63eadde5fc42cd4 (DEFAULT_TABLE table=test-table [id=42b36ce6129d456697af0b9e0c994082]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:15:53.829881 19419 tablet_service.cc:1467] Processing CreateTablet for tablet b90037163b8444b0a63eadde5fc42cd4 (DEFAULT_TABLE table=test-table [id=42b36ce6129d456697af0b9e0c994082]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:15:53.830228 19141 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b90037163b8444b0a63eadde5fc42cd4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:15:53.830514 19419 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b90037163b8444b0a63eadde5fc42cd4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:15:53.831287 19556 tablet_replica.cc:330] T 677eb6aa3951473980bf32e063870661 P 426541981bf84ca1a3e874674a81d687: stopping tablet replica
I20260620 20:15:53.831756 19556 raft_consensus.cc:2229] T 677eb6aa3951473980bf32e063870661 P 426541981bf84ca1a3e874674a81d687 [term 1 LEADER]: Raft consensus shutting down.
I20260620 20:15:53.832652 19556 raft_consensus.cc:2258] T 677eb6aa3951473980bf32e063870661 P 426541981bf84ca1a3e874674a81d687 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:15:53.834671 19556 ts_tablet_manager.cc:1905] T 677eb6aa3951473980bf32e063870661 P 426541981bf84ca1a3e874674a81d687: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:15:53.835700 19510 tablet_bootstrap.cc:492] T b90037163b8444b0a63eadde5fc42cd4 P ce1bf2b4620340b88840348ea13eb73d: Bootstrap starting.
I20260620 20:15:53.837277 19509 tablet_bootstrap.cc:492] T b90037163b8444b0a63eadde5fc42cd4 P 56cd29ec76f24959a145a593ab44b2d0: Bootstrap starting.
I20260620 20:15:53.837314 19510 tablet_bootstrap.cc:654] T b90037163b8444b0a63eadde5fc42cd4 P ce1bf2b4620340b88840348ea13eb73d: Neither blocks nor log segments found. Creating new log.
I20260620 20:15:53.838770 19509 tablet_bootstrap.cc:654] T b90037163b8444b0a63eadde5fc42cd4 P 56cd29ec76f24959a145a593ab44b2d0: Neither blocks nor log segments found. Creating new log.
I20260620 20:15:53.840564 19554 tablet_replica.cc:330] T 677eb6aa3951473980bf32e063870661 P 56cd29ec76f24959a145a593ab44b2d0: stopping tablet replica
I20260620 20:15:53.840902 19509 tablet_bootstrap.cc:492] T b90037163b8444b0a63eadde5fc42cd4 P 56cd29ec76f24959a145a593ab44b2d0: No bootstrap required, opened a new log
I20260620 20:15:53.841061 19509 ts_tablet_manager.cc:1397] T b90037163b8444b0a63eadde5fc42cd4 P 56cd29ec76f24959a145a593ab44b2d0: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260620 20:15:53.841156 19554 raft_consensus.cc:2229] T 677eb6aa3951473980bf32e063870661 P 56cd29ec76f24959a145a593ab44b2d0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 20:15:53.841586 19554 raft_consensus.cc:2258] T 677eb6aa3951473980bf32e063870661 P 56cd29ec76f24959a145a593ab44b2d0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:15:53.841938 19509 raft_consensus.cc:348] T b90037163b8444b0a63eadde5fc42cd4 P 56cd29ec76f24959a145a593ab44b2d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "426541981bf84ca1a3e874674a81d687" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 40783 } } peers { permanent_uuid: "ce1bf2b4620340b88840348ea13eb73d" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 40071 } } peers { permanent_uuid: "56cd29ec76f24959a145a593ab44b2d0" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 45785 } }
I20260620 20:15:53.842195 19509 raft_consensus.cc:374] T b90037163b8444b0a63eadde5fc42cd4 P 56cd29ec76f24959a145a593ab44b2d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:15:53.842283 19509 raft_consensus.cc:729] T b90037163b8444b0a63eadde5fc42cd4 P 56cd29ec76f24959a145a593ab44b2d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56cd29ec76f24959a145a593ab44b2d0, State: Initialized, Role: FOLLOWER
I20260620 20:15:53.842669 19555 tablet_replica.cc:330] T 677eb6aa3951473980bf32e063870661 P ce1bf2b4620340b88840348ea13eb73d: stopping tablet replica
I20260620 20:15:53.842622 19509 consensus_queue.cc:260] T b90037163b8444b0a63eadde5fc42cd4 P 56cd29ec76f24959a145a593ab44b2d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "426541981bf84ca1a3e874674a81d687" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 40783 } } peers { permanent_uuid: "ce1bf2b4620340b88840348ea13eb73d" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 40071 } } peers { permanent_uuid: "56cd29ec76f24959a145a593ab44b2d0" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 45785 } }
I20260620 20:15:53.844054 19554 ts_tablet_manager.cc:1905] T 677eb6aa3951473980bf32e063870661 P 56cd29ec76f24959a145a593ab44b2d0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:15:53.845602 19509 ts_tablet_manager.cc:1428] T b90037163b8444b0a63eadde5fc42cd4 P 56cd29ec76f24959a145a593ab44b2d0: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260620 20:15:53.845996 19556 ts_tablet_manager.cc:1918] T 677eb6aa3951473980bf32e063870661 P 426541981bf84ca1a3e874674a81d687: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 20:15:53.845198 19510 tablet_bootstrap.cc:492] T b90037163b8444b0a63eadde5fc42cd4 P ce1bf2b4620340b88840348ea13eb73d: No bootstrap required, opened a new log
I20260620 20:15:53.846704 19510 ts_tablet_manager.cc:1397] T b90037163b8444b0a63eadde5fc42cd4 P ce1bf2b4620340b88840348ea13eb73d: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.000s
I20260620 20:15:53.846181 19556 log.cc:1196] T 677eb6aa3951473980bf32e063870661 P 426541981bf84ca1a3e874674a81d687: Deleting WAL directory at /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986536687152-17252-0/minicluster-data/ts-0/wal/wals/677eb6aa3951473980bf32e063870661
I20260620 20:15:53.847491 19510 raft_consensus.cc:348] T b90037163b8444b0a63eadde5fc42cd4 P ce1bf2b4620340b88840348ea13eb73d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "426541981bf84ca1a3e874674a81d687" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 40783 } } peers { permanent_uuid: "ce1bf2b4620340b88840348ea13eb73d" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 40071 } } peers { permanent_uuid: "56cd29ec76f24959a145a593ab44b2d0" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 45785 } }
I20260620 20:15:53.847754 19510 raft_consensus.cc:374] T b90037163b8444b0a63eadde5fc42cd4 P ce1bf2b4620340b88840348ea13eb73d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:15:53.847785 19556 ts_tablet_manager.cc:1939] T 677eb6aa3951473980bf32e063870661 P 426541981bf84ca1a3e874674a81d687: Deleting consensus metadata
I20260620 20:15:53.847880 19510 raft_consensus.cc:729] T b90037163b8444b0a63eadde5fc42cd4 P ce1bf2b4620340b88840348ea13eb73d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce1bf2b4620340b88840348ea13eb73d, State: Initialized, Role: FOLLOWER
I20260620 20:15:53.848145 19510 consensus_queue.cc:260] T b90037163b8444b0a63eadde5fc42cd4 P ce1bf2b4620340b88840348ea13eb73d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "426541981bf84ca1a3e874674a81d687" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 40783 } } peers { permanent_uuid: "ce1bf2b4620340b88840348ea13eb73d" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 40071 } } peers { permanent_uuid: "56cd29ec76f24959a145a593ab44b2d0" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 45785 } }
I20260620 20:15:53.848624 19510 ts_tablet_manager.cc:1428] T b90037163b8444b0a63eadde5fc42cd4 P ce1bf2b4620340b88840348ea13eb73d: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260620 20:15:53.852546 19554 ts_tablet_manager.cc:1918] T 677eb6aa3951473980bf32e063870661 P 56cd29ec76f24959a145a593ab44b2d0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 20:15:53.852705 19554 log.cc:1196] T 677eb6aa3951473980bf32e063870661 P 56cd29ec76f24959a145a593ab44b2d0: Deleting WAL directory at /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986536687152-17252-0/minicluster-data/ts-2/wal/wals/677eb6aa3951473980bf32e063870661
I20260620 20:15:53.853106 19554 ts_tablet_manager.cc:1939] T 677eb6aa3951473980bf32e063870661 P 56cd29ec76f24959a145a593ab44b2d0: Deleting consensus metadata
I20260620 20:15:53.854084 19555 raft_consensus.cc:2229] T 677eb6aa3951473980bf32e063870661 P ce1bf2b4620340b88840348ea13eb73d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 20:15:53.854249 19511 tablet_bootstrap.cc:492] T b90037163b8444b0a63eadde5fc42cd4 P 426541981bf84ca1a3e874674a81d687: Bootstrap starting.
I20260620 20:15:53.854473 19555 raft_consensus.cc:2258] T 677eb6aa3951473980bf32e063870661 P ce1bf2b4620340b88840348ea13eb73d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:15:53.854607 18991 catalog_manager.cc:4841] TS 56cd29ec76f24959a145a593ab44b2d0 (127.16.217.3:45785): tablet 677eb6aa3951473980bf32e063870661 (table test-table [id=059007b1134c416698015a3db0bf8dec]) successfully deleted
I20260620 20:15:53.855928 19511 tablet_bootstrap.cc:654] T b90037163b8444b0a63eadde5fc42cd4 P 426541981bf84ca1a3e874674a81d687: Neither blocks nor log segments found. Creating new log.
I20260620 20:15:53.858592 19555 ts_tablet_manager.cc:1905] T 677eb6aa3951473980bf32e063870661 P ce1bf2b4620340b88840348ea13eb73d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:15:53.860841 18988 catalog_manager.cc:4841] TS 426541981bf84ca1a3e874674a81d687 (127.16.217.1:40783): tablet 677eb6aa3951473980bf32e063870661 (table test-table [id=059007b1134c416698015a3db0bf8dec]) successfully deleted
I20260620 20:15:53.861104 19511 tablet_bootstrap.cc:492] T b90037163b8444b0a63eadde5fc42cd4 P 426541981bf84ca1a3e874674a81d687: No bootstrap required, opened a new log
I20260620 20:15:53.861270 19511 ts_tablet_manager.cc:1397] T b90037163b8444b0a63eadde5fc42cd4 P 426541981bf84ca1a3e874674a81d687: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.000s
I20260620 20:15:53.862041 19511 raft_consensus.cc:348] T b90037163b8444b0a63eadde5fc42cd4 P 426541981bf84ca1a3e874674a81d687 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "426541981bf84ca1a3e874674a81d687" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 40783 } } peers { permanent_uuid: "ce1bf2b4620340b88840348ea13eb73d" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 40071 } } peers { permanent_uuid: "56cd29ec76f24959a145a593ab44b2d0" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 45785 } }
I20260620 20:15:53.862299 19511 raft_consensus.cc:374] T b90037163b8444b0a63eadde5fc42cd4 P 426541981bf84ca1a3e874674a81d687 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:15:53.862383 19511 raft_consensus.cc:729] T b90037163b8444b0a63eadde5fc42cd4 P 426541981bf84ca1a3e874674a81d687 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 426541981bf84ca1a3e874674a81d687, State: Initialized, Role: FOLLOWER
I20260620 20:15:53.862607 19511 consensus_queue.cc:260] T b90037163b8444b0a63eadde5fc42cd4 P 426541981bf84ca1a3e874674a81d687 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "426541981bf84ca1a3e874674a81d687" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 40783 } } peers { permanent_uuid: "ce1bf2b4620340b88840348ea13eb73d" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 40071 } } peers { permanent_uuid: "56cd29ec76f24959a145a593ab44b2d0" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 45785 } }
I20260620 20:15:53.863452 19511 ts_tablet_manager.cc:1428] T b90037163b8444b0a63eadde5fc42cd4 P 426541981bf84ca1a3e874674a81d687: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 20:15:53.866541 19555 ts_tablet_manager.cc:1918] T 677eb6aa3951473980bf32e063870661 P ce1bf2b4620340b88840348ea13eb73d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 20:15:53.866731 19555 log.cc:1196] T 677eb6aa3951473980bf32e063870661 P ce1bf2b4620340b88840348ea13eb73d: Deleting WAL directory at /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986536687152-17252-0/minicluster-data/ts-1/wal/wals/677eb6aa3951473980bf32e063870661
I20260620 20:15:53.867328 19555 ts_tablet_manager.cc:1939] T 677eb6aa3951473980bf32e063870661 P ce1bf2b4620340b88840348ea13eb73d: Deleting consensus metadata
I20260620 20:15:53.868846 18988 catalog_manager.cc:4841] TS ce1bf2b4620340b88840348ea13eb73d (127.16.217.2:40071): tablet 677eb6aa3951473980bf32e063870661 (table test-table [id=059007b1134c416698015a3db0bf8dec]) successfully deleted
I20260620 20:15:54.175139 19516 raft_consensus.cc:482] T b90037163b8444b0a63eadde5fc42cd4 P ce1bf2b4620340b88840348ea13eb73d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 20:15:54.175405 19516 raft_consensus.cc:504] T b90037163b8444b0a63eadde5fc42cd4 P ce1bf2b4620340b88840348ea13eb73d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "426541981bf84ca1a3e874674a81d687" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 40783 } } peers { permanent_uuid: "ce1bf2b4620340b88840348ea13eb73d" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 40071 } } peers { permanent_uuid: "56cd29ec76f24959a145a593ab44b2d0" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 45785 } }
I20260620 20:15:54.177040 19516 leader_election.cc:288] T b90037163b8444b0a63eadde5fc42cd4 P ce1bf2b4620340b88840348ea13eb73d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 426541981bf84ca1a3e874674a81d687 (127.16.217.1:40783), 56cd29ec76f24959a145a593ab44b2d0 (127.16.217.3:45785)
I20260620 20:15:54.189023 19536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:54.177519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.217.2:42267 (local address 127.16.217.1:40783)
0620 20:15:54.177789 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0620 20:15:54.177795 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:15:54.178191 (+ 396us) server_negotiation.cc:407] Connection header received
0620 20:15:54.178557 (+ 366us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:15:54.178566 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:15:54.178674 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:15:54.178769 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:15:54.179756 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:54.181410 (+ 1654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:54.185072 (+ 3662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:54.187288 (+ 2216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:54.187460 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:15:54.187584 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0620 20:15:54.188387 (+ 803us) server_negotiation.cc:299] Negotiation successful
0620 20:15:54.188588 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260620 20:15:54.190217 19561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:54.177221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.1:40783 (local address 127.16.217.2:42267)
0620 20:15:54.178048 (+ 827us) negotiation.cc:107] Waiting for socket to connect
0620 20:15:54.178084 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 20:15:54.178341 (+ 257us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:15:54.179077 (+ 736us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:15:54.179086 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:15:54.179124 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:15:54.179507 (+ 383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:15:54.179523 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:15:54.181678 (+ 2155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:15:54.181687 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:15:54.184831 (+ 3144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:15:54.184852 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:15:54.187842 (+ 2990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:15:54.187851 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:15:54.188118 (+ 267us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:15:54.188173 (+ 55us) client_negotiation.cc:769] Sending connection context
0620 20:15:54.189601 (+ 1428us) client_negotiation.cc:240] Negotiation successful
0620 20:15:54.189775 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":671,"thread_start_us":12610,"threads_started":1}
I20260620 20:15:54.191170 19162 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b90037163b8444b0a63eadde5fc42cd4" candidate_uuid: "ce1bf2b4620340b88840348ea13eb73d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "426541981bf84ca1a3e874674a81d687" is_pre_election: true
I20260620 20:15:54.191725 19162 raft_consensus.cc:2454] T b90037163b8444b0a63eadde5fc42cd4 P 426541981bf84ca1a3e874674a81d687 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce1bf2b4620340b88840348ea13eb73d in term 0.
I20260620 20:15:54.192399 19232 leader_election.cc:302] T b90037163b8444b0a63eadde5fc42cd4 P ce1bf2b4620340b88840348ea13eb73d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 426541981bf84ca1a3e874674a81d687, ce1bf2b4620340b88840348ea13eb73d; no voters:
I20260620 20:15:54.192858 19516 raft_consensus.cc:2781] T b90037163b8444b0a63eadde5fc42cd4 P ce1bf2b4620340b88840348ea13eb73d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 20:15:54.193023 19516 raft_consensus.cc:482] T b90037163b8444b0a63eadde5fc42cd4 P ce1bf2b4620340b88840348ea13eb73d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 20:15:54.193126 19516 raft_consensus.cc:3037] T b90037163b8444b0a63eadde5fc42cd4 P ce1bf2b4620340b88840348ea13eb73d [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:15:54.196437 19516 raft_consensus.cc:504] T b90037163b8444b0a63eadde5fc42cd4 P ce1bf2b4620340b88840348ea13eb73d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "426541981bf84ca1a3e874674a81d687" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 40783 } } peers { permanent_uuid: "ce1bf2b4620340b88840348ea13eb73d" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 40071 } } peers { permanent_uuid: "56cd29ec76f24959a145a593ab44b2d0" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 45785 } }
I20260620 20:15:54.196839 19560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:54.177345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.3:45785 (local address 127.16.217.2:35641)
0620 20:15:54.189595 (+ 12250us) negotiation.cc:107] Waiting for socket to connect
0620 20:15:54.189616 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 20:15:54.189752 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:15:54.190759 (+ 1007us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:15:54.190765 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:15:54.190796 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:15:54.191128 (+ 332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:15:54.191139 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:15:54.192695 (+ 1556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:15:54.192700 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:15:54.194717 (+ 2017us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:15:54.194729 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:15:54.195978 (+ 1249us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:15:54.195985 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:15:54.196158 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:15:54.196182 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 20:15:54.196296 (+ 114us) client_negotiation.cc:240] Negotiation successful
0620 20:15:54.196443 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12080,"thread_start_us":707,"threads_started":1}
I20260620 20:15:54.196923 19562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:54.177959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.217.2:35641 (local address 127.16.217.3:45785)
0620 20:15:54.190419 (+ 12460us) server_negotiation.cc:206] Beginning negotiation
0620 20:15:54.190426 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:15:54.190445 (+ 19us) server_negotiation.cc:407] Connection header received
0620 20:15:54.190522 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:15:54.190528 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:15:54.190609 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:15:54.190719 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:15:54.191328 (+ 609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:54.192503 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:54.194897 (+ 2394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:54.195733 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:54.195978 (+ 245us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:15:54.196053 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 20:15:54.196320 (+ 267us) server_negotiation.cc:299] Negotiation successful
0620 20:15:54.196462 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12299,"thread_start_us":12194,"threads_started":1}
I20260620 20:15:54.197584 19516 leader_election.cc:288] T b90037163b8444b0a63eadde5fc42cd4 P ce1bf2b4620340b88840348ea13eb73d [CANDIDATE]: Term 1 election: Requested vote from peers 426541981bf84ca1a3e874674a81d687 (127.16.217.1:40783), 56cd29ec76f24959a145a593ab44b2d0 (127.16.217.3:45785)
I20260620 20:15:54.197554 19444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b90037163b8444b0a63eadde5fc42cd4" candidate_uuid: "ce1bf2b4620340b88840348ea13eb73d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56cd29ec76f24959a145a593ab44b2d0" is_pre_election: true
I20260620 20:15:54.197839 19444 raft_consensus.cc:2454] T b90037163b8444b0a63eadde5fc42cd4 P 56cd29ec76f24959a145a593ab44b2d0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce1bf2b4620340b88840348ea13eb73d in term 0.
I20260620 20:15:54.197930 19162 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b90037163b8444b0a63eadde5fc42cd4" candidate_uuid: "ce1bf2b4620340b88840348ea13eb73d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "426541981bf84ca1a3e874674a81d687"
I20260620 20:15:54.198127 19162 raft_consensus.cc:3037] T b90037163b8444b0a63eadde5fc42cd4 P 426541981bf84ca1a3e874674a81d687 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:15:54.198132 19445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b90037163b8444b0a63eadde5fc42cd4" candidate_uuid: "ce1bf2b4620340b88840348ea13eb73d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56cd29ec76f24959a145a593ab44b2d0"
I20260620 20:15:54.198318 19445 raft_consensus.cc:3037] T b90037163b8444b0a63eadde5fc42cd4 P 56cd29ec76f24959a145a593ab44b2d0 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:15:54.201187 19162 raft_consensus.cc:2454] T b90037163b8444b0a63eadde5fc42cd4 P 426541981bf84ca1a3e874674a81d687 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce1bf2b4620340b88840348ea13eb73d in term 1.
I20260620 20:15:54.201326 19445 raft_consensus.cc:2454] T b90037163b8444b0a63eadde5fc42cd4 P 56cd29ec76f24959a145a593ab44b2d0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce1bf2b4620340b88840348ea13eb73d in term 1.
I20260620 20:15:54.201632 19232 leader_election.cc:302] T b90037163b8444b0a63eadde5fc42cd4 P ce1bf2b4620340b88840348ea13eb73d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 426541981bf84ca1a3e874674a81d687, ce1bf2b4620340b88840348ea13eb73d; no voters:
I20260620 20:15:54.201932 19516 raft_consensus.cc:2781] T b90037163b8444b0a63eadde5fc42cd4 P ce1bf2b4620340b88840348ea13eb73d [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:15:54.202476 19516 raft_consensus.cc:686] T b90037163b8444b0a63eadde5fc42cd4 P ce1bf2b4620340b88840348ea13eb73d [term 1 LEADER]: Becoming Leader. State: Replica: ce1bf2b4620340b88840348ea13eb73d, State: Running, Role: LEADER
I20260620 20:15:54.202890 19516 consensus_queue.cc:237] T b90037163b8444b0a63eadde5fc42cd4 P ce1bf2b4620340b88840348ea13eb73d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "426541981bf84ca1a3e874674a81d687" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 40783 } } peers { permanent_uuid: "ce1bf2b4620340b88840348ea13eb73d" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 40071 } } peers { permanent_uuid: "56cd29ec76f24959a145a593ab44b2d0" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 45785 } }
I20260620 20:15:54.206328 19004 catalog_manager.cc:5495] T b90037163b8444b0a63eadde5fc42cd4 P ce1bf2b4620340b88840348ea13eb73d reported cstate change: term changed from 0 to 1, leader changed from <none> to ce1bf2b4620340b88840348ea13eb73d (127.16.217.2). New cstate: current_term: 1 leader_uuid: "ce1bf2b4620340b88840348ea13eb73d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "426541981bf84ca1a3e874674a81d687" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 40783 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce1bf2b4620340b88840348ea13eb73d" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 40071 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "56cd29ec76f24959a145a593ab44b2d0" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 45785 } health_report { overall_health: UNKNOWN } } }
I20260620 20:15:54.231989 19567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:54.223040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45498 (local address 127.16.217.2:40071)
0620 20:15:54.225647 (+ 2607us) server_negotiation.cc:206] Beginning negotiation
0620 20:15:54.225653 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:15:54.226155 (+ 502us) server_negotiation.cc:407] Connection header received
0620 20:15:54.226400 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:15:54.226404 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:15:54.226482 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:15:54.226678 (+ 196us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 20:15:54.227148 (+ 470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:54.227973 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:54.229364 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:54.229813 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:54.230022 (+ 209us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:15:54.230499 (+ 477us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 20:15:54.231248 (+ 749us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 20:15:54.231403 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0620 20:15:54.231543 (+ 140us) server_negotiation.cc:299] Negotiation successful
0620 20:15:54.231643 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2458,"thread_start_us":2334,"threads_started":1}
I20260620 20:15:54.249203 19536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:54.242912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.217.2:53527 (local address 127.16.217.1:40783)
0620 20:15:54.243101 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0620 20:15:54.243110 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 20:15:54.243594 (+ 484us) server_negotiation.cc:407] Connection header received
0620 20:15:54.243849 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:15:54.243854 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:15:54.243949 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:15:54.244080 (+ 131us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 20:15:54.244616 (+ 536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:54.245487 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:54.247243 (+ 1756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:54.247786 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:54.247936 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:15:54.248418 (+ 482us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 20:15:54.248526 (+ 108us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 20:15:54.248635 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0620 20:15:54.248749 (+ 114us) server_negotiation.cc:299] Negotiation successful
0620 20:15:54.248853 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 20:15:54.249262 19570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:54.242713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.1:40783 (local address 127.16.217.2:53527)
0620 20:15:54.243486 (+ 773us) negotiation.cc:107] Waiting for socket to connect
0620 20:15:54.243502 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 20:15:54.243660 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:15:54.244110 (+ 450us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:15:54.244115 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:15:54.244127 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 20:15:54.244449 (+ 322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:15:54.244458 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:15:54.245636 (+ 1178us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:15:54.245642 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:15:54.247073 (+ 1431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:15:54.247085 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:15:54.247992 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:15:54.248002 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:15:54.248165 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:15:54.248223 (+ 58us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 20:15:54.248632 (+ 409us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 20:15:54.248645 (+ 13us) client_negotiation.cc:769] Sending connection context
0620 20:15:54.248731 (+ 86us) client_negotiation.cc:240] Negotiation successful
0620 20:15:54.248868 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":636,"thread_start_us":425,"threads_started":1}
I20260620 20:15:54.259341 19445 raft_consensus.cc:1261] T b90037163b8444b0a63eadde5fc42cd4 P 56cd29ec76f24959a145a593ab44b2d0 [term 1 FOLLOWER]: Refusing update from remote peer ce1bf2b4620340b88840348ea13eb73d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:15:54.259997 19162 raft_consensus.cc:1261] T b90037163b8444b0a63eadde5fc42cd4 P 426541981bf84ca1a3e874674a81d687 [term 1 FOLLOWER]: Refusing update from remote peer ce1bf2b4620340b88840348ea13eb73d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:15:54.260888 19516 consensus_queue.cc:1035] T b90037163b8444b0a63eadde5fc42cd4 P ce1bf2b4620340b88840348ea13eb73d [LEADER]: Connected to new peer: Peer: permanent_uuid: "426541981bf84ca1a3e874674a81d687" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 40783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:15:54.261482 19563 consensus_queue.cc:1035] T b90037163b8444b0a63eadde5fc42cd4 P ce1bf2b4620340b88840348ea13eb73d [LEADER]: Connected to new peer: Peer: permanent_uuid: "56cd29ec76f24959a145a593ab44b2d0" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 45785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:15:54.268435 19538 mvcc.cc:205] Tried to move back new op lower bound from 7299016926239547392 to 7299016926018482176. Current Snapshot: MvccSnapshot[applied={T|T < 7299016926239547392}]
I20260620 20:15:54.296490 19567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:54.289744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.217.1:36843 (local address 127.16.217.2:40071)
0620 20:15:54.289924 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0620 20:15:54.289931 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:15:54.290407 (+ 476us) server_negotiation.cc:407] Connection header received
0620 20:15:54.290622 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:15:54.290628 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:15:54.290707 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:15:54.290824 (+ 117us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 20:15:54.291572 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:54.292743 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:54.294527 (+ 1784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:54.294976 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:54.295127 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:15:54.295621 (+ 494us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 20:15:54.295754 (+ 133us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 20:15:54.295824 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 20:15:54.296064 (+ 240us) server_negotiation.cc:299] Negotiation successful
0620 20:15:54.296149 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 20:15:54.297516 19575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:54.289407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.2:40071 (local address 127.16.217.1:36843)
0620 20:15:54.290298 (+ 891us) negotiation.cc:107] Waiting for socket to connect
0620 20:15:54.290316 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 20:15:54.290475 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:15:54.291064 (+ 589us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:15:54.291070 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:15:54.291083 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 20:15:54.291363 (+ 280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:15:54.291378 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:15:54.292943 (+ 1565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:15:54.292950 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:15:54.294379 (+ 1429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:15:54.294392 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:15:54.295138 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:15:54.295144 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:15:54.295359 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:15:54.295408 (+ 49us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 20:15:54.295914 (+ 506us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 20:15:54.295925 (+ 11us) client_negotiation.cc:769] Sending connection context
0620 20:15:54.296945 (+ 1020us) client_negotiation.cc:240] Negotiation successful
0620 20:15:54.297095 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":739,"thread_start_us":595,"threads_started":1}
W20260620 20:15:54.298719 19538 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260620 20:15:55.939000 19004 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55264:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 20:15:55.939194 19004 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55264:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 20:15:55.943490 19004 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ebfe81c8d02342e6bc65b45342cd2b9d: Sending DeleteTablet for 3 replicas of tablet b90037163b8444b0a63eadde5fc42cd4
I20260620 20:15:55.944249 19141 tablet_service.cc:1514] Processing DeleteTablet for tablet b90037163b8444b0a63eadde5fc42cd4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:15:55 UTC) from {username='kudu', principal='kudu/127.16.217.62@KRBTEST.COM'} at 127.0.0.1:38800
I20260620 20:15:55.944528 19419 tablet_service.cc:1514] Processing DeleteTablet for tablet b90037163b8444b0a63eadde5fc42cd4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:15:55 UTC) from {username='kudu', principal='kudu/127.16.217.62@KRBTEST.COM'} at 127.0.0.1:41296
I20260620 20:15:55.946269 19281 tablet_service.cc:1514] Processing DeleteTablet for tablet b90037163b8444b0a63eadde5fc42cd4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:15:55 UTC) from {username='kudu', principal='kudu/127.16.217.62@KRBTEST.COM'} at 127.0.0.1:45486
I20260620 20:15:55.958235 19601 tablet_replica.cc:330] T b90037163b8444b0a63eadde5fc42cd4 P 426541981bf84ca1a3e874674a81d687: stopping tablet replica
I20260620 20:15:55.958475 19601 raft_consensus.cc:2229] T b90037163b8444b0a63eadde5fc42cd4 P 426541981bf84ca1a3e874674a81d687 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 20:15:55.958722 19601 raft_consensus.cc:2258] T b90037163b8444b0a63eadde5fc42cd4 P 426541981bf84ca1a3e874674a81d687 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:15:55.960083 19601 ts_tablet_manager.cc:1905] T b90037163b8444b0a63eadde5fc42cd4 P 426541981bf84ca1a3e874674a81d687: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:15:55.966004 19601 ts_tablet_manager.cc:1918] T b90037163b8444b0a63eadde5fc42cd4 P 426541981bf84ca1a3e874674a81d687: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 20:15:55.966301 19601 log.cc:1196] T b90037163b8444b0a63eadde5fc42cd4 P 426541981bf84ca1a3e874674a81d687: Deleting WAL directory at /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986536687152-17252-0/minicluster-data/ts-0/wal/wals/b90037163b8444b0a63eadde5fc42cd4
I20260620 20:15:55.966974 19601 ts_tablet_manager.cc:1939] T b90037163b8444b0a63eadde5fc42cd4 P 426541981bf84ca1a3e874674a81d687: Deleting consensus metadata
I20260620 20:15:55.968618 19603 tablet_replica.cc:330] T b90037163b8444b0a63eadde5fc42cd4 P ce1bf2b4620340b88840348ea13eb73d: stopping tablet replica
I20260620 20:15:55.968555 18988 catalog_manager.cc:4841] TS 426541981bf84ca1a3e874674a81d687 (127.16.217.1:40783): tablet b90037163b8444b0a63eadde5fc42cd4 (table test-table [id=42b36ce6129d456697af0b9e0c994082]) successfully deleted
I20260620 20:15:55.968945 19603 raft_consensus.cc:2229] T b90037163b8444b0a63eadde5fc42cd4 P ce1bf2b4620340b88840348ea13eb73d [term 1 LEADER]: Raft consensus shutting down.
I20260620 20:15:55.969645 19603 raft_consensus.cc:2258] T b90037163b8444b0a63eadde5fc42cd4 P ce1bf2b4620340b88840348ea13eb73d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:15:55.970616 19602 tablet_replica.cc:330] T b90037163b8444b0a63eadde5fc42cd4 P 56cd29ec76f24959a145a593ab44b2d0: stopping tablet replica
I20260620 20:15:55.970906 19603 ts_tablet_manager.cc:1905] T b90037163b8444b0a63eadde5fc42cd4 P ce1bf2b4620340b88840348ea13eb73d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:15:55.970873 19602 raft_consensus.cc:2229] T b90037163b8444b0a63eadde5fc42cd4 P 56cd29ec76f24959a145a593ab44b2d0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 20:15:55.971166 19602 raft_consensus.cc:2258] T b90037163b8444b0a63eadde5fc42cd4 P 56cd29ec76f24959a145a593ab44b2d0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:15:55.972092 19602 ts_tablet_manager.cc:1905] T b90037163b8444b0a63eadde5fc42cd4 P 56cd29ec76f24959a145a593ab44b2d0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:15:55.977921 19603 ts_tablet_manager.cc:1918] T b90037163b8444b0a63eadde5fc42cd4 P ce1bf2b4620340b88840348ea13eb73d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 20:15:55.978137 19603 log.cc:1196] T b90037163b8444b0a63eadde5fc42cd4 P ce1bf2b4620340b88840348ea13eb73d: Deleting WAL directory at /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986536687152-17252-0/minicluster-data/ts-1/wal/wals/b90037163b8444b0a63eadde5fc42cd4
I20260620 20:15:55.978685 19603 ts_tablet_manager.cc:1939] T b90037163b8444b0a63eadde5fc42cd4 P ce1bf2b4620340b88840348ea13eb73d: Deleting consensus metadata
I20260620 20:15:55.979236 19602 ts_tablet_manager.cc:1918] T b90037163b8444b0a63eadde5fc42cd4 P 56cd29ec76f24959a145a593ab44b2d0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 20:15:55.979432 19602 log.cc:1196] T b90037163b8444b0a63eadde5fc42cd4 P 56cd29ec76f24959a145a593ab44b2d0: Deleting WAL directory at /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986536687152-17252-0/minicluster-data/ts-2/wal/wals/b90037163b8444b0a63eadde5fc42cd4
I20260620 20:15:55.980098 18988 catalog_manager.cc:4841] TS ce1bf2b4620340b88840348ea13eb73d (127.16.217.2:40071): tablet b90037163b8444b0a63eadde5fc42cd4 (table test-table [id=42b36ce6129d456697af0b9e0c994082]) successfully deleted
I20260620 20:15:55.980306 19602 ts_tablet_manager.cc:1939] T b90037163b8444b0a63eadde5fc42cd4 P 56cd29ec76f24959a145a593ab44b2d0: Deleting consensus metadata
I20260620 20:15:55.981370 18991 catalog_manager.cc:4841] TS 56cd29ec76f24959a145a593ab44b2d0 (127.16.217.3:45785): tablet b90037163b8444b0a63eadde5fc42cd4 (table test-table [id=42b36ce6129d456697af0b9e0c994082]) successfully deleted
Jun 20 20:15:55 dist-test-slave-2hpg krb5kdc[18959](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986555, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 20:15:56 dist-test-slave-2hpg krb5kdc[18959](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986555, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.16.217.62@KRBTEST.COM
I20260620 20:15:56.022511 19611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:56.002129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55292 (local address 127.16.217.62:38007)
0620 20:15:56.002714 (+ 585us) server_negotiation.cc:206] Beginning negotiation
0620 20:15:56.002722 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 20:15:56.002834 (+ 112us) server_negotiation.cc:407] Connection header received
0620 20:15:56.003074 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:15:56.003078 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:15:56.003146 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:15:56.003258 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:15:56.004590 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:56.005379 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:56.006720 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:56.007144 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:56.007285 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:15:56.017037 (+ 9752us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:15:56.017043 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:15:56.017046 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:15:56.017086 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:15:56.020188 (+ 3102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:15:56.020840 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:15:56.020845 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:15:56.020847 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:15:56.020966 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:15:56.021266 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:15:56.021269 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:15:56.021271 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:15:56.021570 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:15:56.021781 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0620 20:15:56.022060 (+ 279us) server_negotiation.cc:299] Negotiation successful
0620 20:15:56.022246 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":437,"thread_start_us":272,"threads_started":1}
I20260620 20:15:56.026034 19004 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55292:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 20:15:56.026691 19004 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 20:15:56.033911 19141 tablet_service.cc:1467] Processing CreateTablet for tablet b3080185d0684d02a0f30cac2990970f (DEFAULT_TABLE table=test-table [id=7df8686f6fdb4a0ab6ba14023204e6a0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:15:56.034093 19419 tablet_service.cc:1467] Processing CreateTablet for tablet b3080185d0684d02a0f30cac2990970f (DEFAULT_TABLE table=test-table [id=7df8686f6fdb4a0ab6ba14023204e6a0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:15:56.034420 19141 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b3080185d0684d02a0f30cac2990970f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:15:56.034330 19281 tablet_service.cc:1467] Processing CreateTablet for tablet b3080185d0684d02a0f30cac2990970f (DEFAULT_TABLE table=test-table [id=7df8686f6fdb4a0ab6ba14023204e6a0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:15:56.034531 19419 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b3080185d0684d02a0f30cac2990970f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:15:56.034758 19281 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b3080185d0684d02a0f30cac2990970f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:15:56.040469 19612 tablet_bootstrap.cc:492] T b3080185d0684d02a0f30cac2990970f P ce1bf2b4620340b88840348ea13eb73d: Bootstrap starting.
I20260620 20:15:56.042426 19614 tablet_bootstrap.cc:492] T b3080185d0684d02a0f30cac2990970f P 426541981bf84ca1a3e874674a81d687: Bootstrap starting.
I20260620 20:15:56.042598 19612 tablet_bootstrap.cc:654] T b3080185d0684d02a0f30cac2990970f P ce1bf2b4620340b88840348ea13eb73d: Neither blocks nor log segments found. Creating new log.
I20260620 20:15:56.043087 19613 tablet_bootstrap.cc:492] T b3080185d0684d02a0f30cac2990970f P 56cd29ec76f24959a145a593ab44b2d0: Bootstrap starting.
I20260620 20:15:56.044314 19613 tablet_bootstrap.cc:654] T b3080185d0684d02a0f30cac2990970f P 56cd29ec76f24959a145a593ab44b2d0: Neither blocks nor log segments found. Creating new log.
I20260620 20:15:56.044929 19614 tablet_bootstrap.cc:654] T b3080185d0684d02a0f30cac2990970f P 426541981bf84ca1a3e874674a81d687: Neither blocks nor log segments found. Creating new log.
I20260620 20:15:56.047890 19614 tablet_bootstrap.cc:492] T b3080185d0684d02a0f30cac2990970f P 426541981bf84ca1a3e874674a81d687: No bootstrap required, opened a new log
I20260620 20:15:56.048137 19614 ts_tablet_manager.cc:1397] T b3080185d0684d02a0f30cac2990970f P 426541981bf84ca1a3e874674a81d687: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260620 20:15:56.049295 19614 raft_consensus.cc:348] T b3080185d0684d02a0f30cac2990970f P 426541981bf84ca1a3e874674a81d687 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "426541981bf84ca1a3e874674a81d687" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 40783 } } peers { permanent_uuid: "ce1bf2b4620340b88840348ea13eb73d" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 40071 } } peers { permanent_uuid: "56cd29ec76f24959a145a593ab44b2d0" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 45785 } }
I20260620 20:15:56.049705 19614 raft_consensus.cc:374] T b3080185d0684d02a0f30cac2990970f P 426541981bf84ca1a3e874674a81d687 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:15:56.049886 19614 raft_consensus.cc:729] T b3080185d0684d02a0f30cac2990970f P 426541981bf84ca1a3e874674a81d687 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 426541981bf84ca1a3e874674a81d687, State: Initialized, Role: FOLLOWER
I20260620 20:15:56.050328 19614 consensus_queue.cc:260] T b3080185d0684d02a0f30cac2990970f P 426541981bf84ca1a3e874674a81d687 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "426541981bf84ca1a3e874674a81d687" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 40783 } } peers { permanent_uuid: "ce1bf2b4620340b88840348ea13eb73d" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 40071 } } peers { permanent_uuid: "56cd29ec76f24959a145a593ab44b2d0" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 45785 } }
I20260620 20:15:56.050964 19613 tablet_bootstrap.cc:492] T b3080185d0684d02a0f30cac2990970f P 56cd29ec76f24959a145a593ab44b2d0: No bootstrap required, opened a new log
I20260620 20:15:56.051060 19613 ts_tablet_manager.cc:1397] T b3080185d0684d02a0f30cac2990970f P 56cd29ec76f24959a145a593ab44b2d0: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260620 20:15:56.051667 19614 ts_tablet_manager.cc:1428] T b3080185d0684d02a0f30cac2990970f P 426541981bf84ca1a3e874674a81d687: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 20:15:56.052011 19613 raft_consensus.cc:348] T b3080185d0684d02a0f30cac2990970f P 56cd29ec76f24959a145a593ab44b2d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "426541981bf84ca1a3e874674a81d687" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 40783 } } peers { permanent_uuid: "ce1bf2b4620340b88840348ea13eb73d" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 40071 } } peers { permanent_uuid: "56cd29ec76f24959a145a593ab44b2d0" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 45785 } }
I20260620 20:15:56.052187 19612 tablet_bootstrap.cc:492] T b3080185d0684d02a0f30cac2990970f P ce1bf2b4620340b88840348ea13eb73d: No bootstrap required, opened a new log
I20260620 20:15:56.052264 19613 raft_consensus.cc:374] T b3080185d0684d02a0f30cac2990970f P 56cd29ec76f24959a145a593ab44b2d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:15:56.052326 19612 ts_tablet_manager.cc:1397] T b3080185d0684d02a0f30cac2990970f P ce1bf2b4620340b88840348ea13eb73d: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.007s
I20260620 20:15:56.052368 19613 raft_consensus.cc:729] T b3080185d0684d02a0f30cac2990970f P 56cd29ec76f24959a145a593ab44b2d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56cd29ec76f24959a145a593ab44b2d0, State: Initialized, Role: FOLLOWER
I20260620 20:15:56.052608 19613 consensus_queue.cc:260] T b3080185d0684d02a0f30cac2990970f P 56cd29ec76f24959a145a593ab44b2d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "426541981bf84ca1a3e874674a81d687" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 40783 } } peers { permanent_uuid: "ce1bf2b4620340b88840348ea13eb73d" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 40071 } } peers { permanent_uuid: "56cd29ec76f24959a145a593ab44b2d0" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 45785 } }
I20260620 20:15:56.053187 19612 raft_consensus.cc:348] T b3080185d0684d02a0f30cac2990970f P ce1bf2b4620340b88840348ea13eb73d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "426541981bf84ca1a3e874674a81d687" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 40783 } } peers { permanent_uuid: "ce1bf2b4620340b88840348ea13eb73d" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 40071 } } peers { permanent_uuid: "56cd29ec76f24959a145a593ab44b2d0" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 45785 } }
I20260620 20:15:56.053385 19613 ts_tablet_manager.cc:1428] T b3080185d0684d02a0f30cac2990970f P 56cd29ec76f24959a145a593ab44b2d0: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 20:15:56.053488 19612 raft_consensus.cc:374] T b3080185d0684d02a0f30cac2990970f P ce1bf2b4620340b88840348ea13eb73d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:15:56.053612 19612 raft_consensus.cc:729] T b3080185d0684d02a0f30cac2990970f P ce1bf2b4620340b88840348ea13eb73d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce1bf2b4620340b88840348ea13eb73d, State: Initialized, Role: FOLLOWER
I20260620 20:15:56.054127 19612 consensus_queue.cc:260] T b3080185d0684d02a0f30cac2990970f P ce1bf2b4620340b88840348ea13eb73d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "426541981bf84ca1a3e874674a81d687" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 40783 } } peers { permanent_uuid: "ce1bf2b4620340b88840348ea13eb73d" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 40071 } } peers { permanent_uuid: "56cd29ec76f24959a145a593ab44b2d0" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 45785 } }
I20260620 20:15:56.054915 19612 ts_tablet_manager.cc:1428] T b3080185d0684d02a0f30cac2990970f P ce1bf2b4620340b88840348ea13eb73d: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260620 20:15:56.103482 19579 raft_consensus.cc:482] T b3080185d0684d02a0f30cac2990970f P ce1bf2b4620340b88840348ea13eb73d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 20:15:56.103679 19579 raft_consensus.cc:504] T b3080185d0684d02a0f30cac2990970f P ce1bf2b4620340b88840348ea13eb73d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "426541981bf84ca1a3e874674a81d687" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 40783 } } peers { permanent_uuid: "ce1bf2b4620340b88840348ea13eb73d" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 40071 } } peers { permanent_uuid: "56cd29ec76f24959a145a593ab44b2d0" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 45785 } }
I20260620 20:15:56.104424 19579 leader_election.cc:288] T b3080185d0684d02a0f30cac2990970f P ce1bf2b4620340b88840348ea13eb73d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 426541981bf84ca1a3e874674a81d687 (127.16.217.1:40783), 56cd29ec76f24959a145a593ab44b2d0 (127.16.217.3:45785)
I20260620 20:15:56.105010 19444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3080185d0684d02a0f30cac2990970f" candidate_uuid: "ce1bf2b4620340b88840348ea13eb73d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56cd29ec76f24959a145a593ab44b2d0" is_pre_election: true
I20260620 20:15:56.105010 19162 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3080185d0684d02a0f30cac2990970f" candidate_uuid: "ce1bf2b4620340b88840348ea13eb73d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "426541981bf84ca1a3e874674a81d687" is_pre_election: true
I20260620 20:15:56.105257 19444 raft_consensus.cc:2454] T b3080185d0684d02a0f30cac2990970f P 56cd29ec76f24959a145a593ab44b2d0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce1bf2b4620340b88840348ea13eb73d in term 0.
I20260620 20:15:56.105274 19162 raft_consensus.cc:2454] T b3080185d0684d02a0f30cac2990970f P 426541981bf84ca1a3e874674a81d687 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce1bf2b4620340b88840348ea13eb73d in term 0.
I20260620 20:15:56.105758 19232 leader_election.cc:302] T b3080185d0684d02a0f30cac2990970f P ce1bf2b4620340b88840348ea13eb73d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 426541981bf84ca1a3e874674a81d687, 56cd29ec76f24959a145a593ab44b2d0, ce1bf2b4620340b88840348ea13eb73d; no voters:
I20260620 20:15:56.106076 19579 raft_consensus.cc:2781] T b3080185d0684d02a0f30cac2990970f P ce1bf2b4620340b88840348ea13eb73d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 20:15:56.106205 19579 raft_consensus.cc:482] T b3080185d0684d02a0f30cac2990970f P ce1bf2b4620340b88840348ea13eb73d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 20:15:56.106319 19579 raft_consensus.cc:3037] T b3080185d0684d02a0f30cac2990970f P ce1bf2b4620340b88840348ea13eb73d [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:15:56.108336 19579 raft_consensus.cc:504] T b3080185d0684d02a0f30cac2990970f P ce1bf2b4620340b88840348ea13eb73d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "426541981bf84ca1a3e874674a81d687" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 40783 } } peers { permanent_uuid: "ce1bf2b4620340b88840348ea13eb73d" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 40071 } } peers { permanent_uuid: "56cd29ec76f24959a145a593ab44b2d0" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 45785 } }
I20260620 20:15:56.108861 19579 leader_election.cc:288] T b3080185d0684d02a0f30cac2990970f P ce1bf2b4620340b88840348ea13eb73d [CANDIDATE]: Term 1 election: Requested vote from peers 426541981bf84ca1a3e874674a81d687 (127.16.217.1:40783), 56cd29ec76f24959a145a593ab44b2d0 (127.16.217.3:45785)
I20260620 20:15:56.109319 19162 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3080185d0684d02a0f30cac2990970f" candidate_uuid: "ce1bf2b4620340b88840348ea13eb73d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "426541981bf84ca1a3e874674a81d687"
I20260620 20:15:56.109354 19444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3080185d0684d02a0f30cac2990970f" candidate_uuid: "ce1bf2b4620340b88840348ea13eb73d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56cd29ec76f24959a145a593ab44b2d0"
I20260620 20:15:56.109485 19162 raft_consensus.cc:3037] T b3080185d0684d02a0f30cac2990970f P 426541981bf84ca1a3e874674a81d687 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:15:56.109548 19444 raft_consensus.cc:3037] T b3080185d0684d02a0f30cac2990970f P 56cd29ec76f24959a145a593ab44b2d0 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:15:56.111470 19162 raft_consensus.cc:2454] T b3080185d0684d02a0f30cac2990970f P 426541981bf84ca1a3e874674a81d687 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce1bf2b4620340b88840348ea13eb73d in term 1.
I20260620 20:15:56.111492 19444 raft_consensus.cc:2454] T b3080185d0684d02a0f30cac2990970f P 56cd29ec76f24959a145a593ab44b2d0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce1bf2b4620340b88840348ea13eb73d in term 1.
I20260620 20:15:56.111851 19232 leader_election.cc:302] T b3080185d0684d02a0f30cac2990970f P ce1bf2b4620340b88840348ea13eb73d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 426541981bf84ca1a3e874674a81d687, ce1bf2b4620340b88840348ea13eb73d; no voters:
I20260620 20:15:56.112170 19579 raft_consensus.cc:2781] T b3080185d0684d02a0f30cac2990970f P ce1bf2b4620340b88840348ea13eb73d [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:15:56.112329 19579 raft_consensus.cc:686] T b3080185d0684d02a0f30cac2990970f P ce1bf2b4620340b88840348ea13eb73d [term 1 LEADER]: Becoming Leader. State: Replica: ce1bf2b4620340b88840348ea13eb73d, State: Running, Role: LEADER
I20260620 20:15:56.112588 19579 consensus_queue.cc:237] T b3080185d0684d02a0f30cac2990970f P ce1bf2b4620340b88840348ea13eb73d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "426541981bf84ca1a3e874674a81d687" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 40783 } } peers { permanent_uuid: "ce1bf2b4620340b88840348ea13eb73d" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 40071 } } peers { permanent_uuid: "56cd29ec76f24959a145a593ab44b2d0" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 45785 } }
I20260620 20:15:56.115801 19004 catalog_manager.cc:5495] T b3080185d0684d02a0f30cac2990970f P ce1bf2b4620340b88840348ea13eb73d reported cstate change: term changed from 0 to 1, leader changed from <none> to ce1bf2b4620340b88840348ea13eb73d (127.16.217.2). New cstate: current_term: 1 leader_uuid: "ce1bf2b4620340b88840348ea13eb73d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "426541981bf84ca1a3e874674a81d687" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 40783 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce1bf2b4620340b88840348ea13eb73d" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 40071 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "56cd29ec76f24959a145a593ab44b2d0" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 45785 } health_report { overall_health: UNKNOWN } } }
I20260620 20:15:56.152282 19621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:56.147334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45506 (local address 127.16.217.2:40071)
0620 20:15:56.147867 (+ 533us) server_negotiation.cc:206] Beginning negotiation
0620 20:15:56.147871 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:15:56.147888 (+ 17us) server_negotiation.cc:407] Connection header received
0620 20:15:56.147933 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:15:56.147937 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:15:56.147993 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:15:56.148099 (+ 106us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 20:15:56.148483 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:56.149261 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:56.150604 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:56.151027 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:56.151163 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:15:56.151554 (+ 391us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 20:15:56.151671 (+ 117us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 20:15:56.151769 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0620 20:15:56.151916 (+ 147us) server_negotiation.cc:299] Negotiation successful
0620 20:15:56.151987 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":400,"thread_start_us":273,"threads_started":1}
I20260620 20:15:56.161156 19162 raft_consensus.cc:1261] T b3080185d0684d02a0f30cac2990970f P 426541981bf84ca1a3e874674a81d687 [term 1 FOLLOWER]: Refusing update from remote peer ce1bf2b4620340b88840348ea13eb73d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:15:56.161195 19444 raft_consensus.cc:1261] T b3080185d0684d02a0f30cac2990970f P 56cd29ec76f24959a145a593ab44b2d0 [term 1 FOLLOWER]: Refusing update from remote peer ce1bf2b4620340b88840348ea13eb73d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:15:56.161652 19579 consensus_queue.cc:1035] T b3080185d0684d02a0f30cac2990970f P ce1bf2b4620340b88840348ea13eb73d [LEADER]: Connected to new peer: Peer: permanent_uuid: "426541981bf84ca1a3e874674a81d687" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 40783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:15:56.161993 19590 consensus_queue.cc:1035] T b3080185d0684d02a0f30cac2990970f P ce1bf2b4620340b88840348ea13eb73d [LEADER]: Connected to new peer: Peer: permanent_uuid: "56cd29ec76f24959a145a593ab44b2d0" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 45785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:15:56.220528 19003 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55292:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 20:15:56.220803 19003 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55292:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 20:15:56.223678 19003 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ebfe81c8d02342e6bc65b45342cd2b9d: Sending DeleteTablet for 3 replicas of tablet b3080185d0684d02a0f30cac2990970f
I20260620 20:15:56.224409 19141 tablet_service.cc:1514] Processing DeleteTablet for tablet b3080185d0684d02a0f30cac2990970f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:15:56 UTC) from {username='kudu', principal='kudu/127.16.217.62@KRBTEST.COM'} at 127.0.0.1:38800
I20260620 20:15:56.224416 19281 tablet_service.cc:1514] Processing DeleteTablet for tablet b3080185d0684d02a0f30cac2990970f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:15:56 UTC) from {username='kudu', principal='kudu/127.16.217.62@KRBTEST.COM'} at 127.0.0.1:45486
I20260620 20:15:56.224556 19419 tablet_service.cc:1514] Processing DeleteTablet for tablet b3080185d0684d02a0f30cac2990970f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:15:56 UTC) from {username='kudu', principal='kudu/127.16.217.62@KRBTEST.COM'} at 127.0.0.1:41296
I20260620 20:15:56.224686 19601 tablet_replica.cc:330] T b3080185d0684d02a0f30cac2990970f P 426541981bf84ca1a3e874674a81d687: stopping tablet replica
I20260620 20:15:56.224875 19602 tablet_replica.cc:330] T b3080185d0684d02a0f30cac2990970f P 56cd29ec76f24959a145a593ab44b2d0: stopping tablet replica
I20260620 20:15:56.224949 19603 tablet_replica.cc:330] T b3080185d0684d02a0f30cac2990970f P ce1bf2b4620340b88840348ea13eb73d: stopping tablet replica
I20260620 20:15:56.224946 19601 raft_consensus.cc:2229] T b3080185d0684d02a0f30cac2990970f P 426541981bf84ca1a3e874674a81d687 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 20:15:56.225062 19602 raft_consensus.cc:2229] T b3080185d0684d02a0f30cac2990970f P 56cd29ec76f24959a145a593ab44b2d0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 20:15:56.225123 19601 raft_consensus.cc:2258] T b3080185d0684d02a0f30cac2990970f P 426541981bf84ca1a3e874674a81d687 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:15:56.225148 19603 raft_consensus.cc:2229] T b3080185d0684d02a0f30cac2990970f P ce1bf2b4620340b88840348ea13eb73d [term 1 LEADER]: Raft consensus shutting down.
I20260620 20:15:56.225234 19602 raft_consensus.cc:2258] T b3080185d0684d02a0f30cac2990970f P 56cd29ec76f24959a145a593ab44b2d0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:15:56.225507 19603 raft_consensus.cc:2258] T b3080185d0684d02a0f30cac2990970f P ce1bf2b4620340b88840348ea13eb73d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:15:56.225854 19003 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55292:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260620 20:15:56.226583 19603 ts_tablet_manager.cc:1905] T b3080185d0684d02a0f30cac2990970f P ce1bf2b4620340b88840348ea13eb73d: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260620 20:15:56.226605 19003 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 20:15:56.226732 19602 ts_tablet_manager.cc:1905] T b3080185d0684d02a0f30cac2990970f P 56cd29ec76f24959a145a593ab44b2d0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:15:56.226742 19601 ts_tablet_manager.cc:1905] T b3080185d0684d02a0f30cac2990970f P 426541981bf84ca1a3e874674a81d687: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:15:56.233482 19601 ts_tablet_manager.cc:1918] T b3080185d0684d02a0f30cac2990970f P 426541981bf84ca1a3e874674a81d687: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 20:15:56.233536 19602 ts_tablet_manager.cc:1918] T b3080185d0684d02a0f30cac2990970f P 56cd29ec76f24959a145a593ab44b2d0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 20:15:56.233688 19601 log.cc:1196] T b3080185d0684d02a0f30cac2990970f P 426541981bf84ca1a3e874674a81d687: Deleting WAL directory at /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986536687152-17252-0/minicluster-data/ts-0/wal/wals/b3080185d0684d02a0f30cac2990970f
I20260620 20:15:56.233701 19602 log.cc:1196] T b3080185d0684d02a0f30cac2990970f P 56cd29ec76f24959a145a593ab44b2d0: Deleting WAL directory at /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986536687152-17252-0/minicluster-data/ts-2/wal/wals/b3080185d0684d02a0f30cac2990970f
I20260620 20:15:56.233922 19281 tablet_service.cc:1467] Processing CreateTablet for tablet 3890388daa25434993b6d6bbed1f0fd0 (DEFAULT_TABLE table=test-table [id=672c70e29c704ff6a2671cbbf2e10089]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:15:56.233933 19141 tablet_service.cc:1467] Processing CreateTablet for tablet 3890388daa25434993b6d6bbed1f0fd0 (DEFAULT_TABLE table=test-table [id=672c70e29c704ff6a2671cbbf2e10089]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:15:56.234170 19602 ts_tablet_manager.cc:1939] T b3080185d0684d02a0f30cac2990970f P 56cd29ec76f24959a145a593ab44b2d0: Deleting consensus metadata
I20260620 20:15:56.234203 19601 ts_tablet_manager.cc:1939] T b3080185d0684d02a0f30cac2990970f P 426541981bf84ca1a3e874674a81d687: Deleting consensus metadata
I20260620 20:15:56.234323 19141 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3890388daa25434993b6d6bbed1f0fd0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:15:56.234345 19281 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3890388daa25434993b6d6bbed1f0fd0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:15:56.235298 19603 ts_tablet_manager.cc:1918] T b3080185d0684d02a0f30cac2990970f P ce1bf2b4620340b88840348ea13eb73d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 20:15:56.235500 19603 log.cc:1196] T b3080185d0684d02a0f30cac2990970f P ce1bf2b4620340b88840348ea13eb73d: Deleting WAL directory at /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986536687152-17252-0/minicluster-data/ts-1/wal/wals/b3080185d0684d02a0f30cac2990970f
I20260620 20:15:56.236202 19603 ts_tablet_manager.cc:1939] T b3080185d0684d02a0f30cac2990970f P ce1bf2b4620340b88840348ea13eb73d: Deleting consensus metadata
I20260620 20:15:56.237183 18988 catalog_manager.cc:4841] TS ce1bf2b4620340b88840348ea13eb73d (127.16.217.2:40071): tablet b3080185d0684d02a0f30cac2990970f (table test-table [id=7df8686f6fdb4a0ab6ba14023204e6a0]) successfully deleted
I20260620 20:15:56.237457 18988 catalog_manager.cc:4841] TS 426541981bf84ca1a3e874674a81d687 (127.16.217.1:40783): tablet b3080185d0684d02a0f30cac2990970f (table test-table [id=7df8686f6fdb4a0ab6ba14023204e6a0]) successfully deleted
I20260620 20:15:56.237797 18991 catalog_manager.cc:4841] TS 56cd29ec76f24959a145a593ab44b2d0 (127.16.217.3:45785): tablet b3080185d0684d02a0f30cac2990970f (table test-table [id=7df8686f6fdb4a0ab6ba14023204e6a0]) successfully deleted
I20260620 20:15:56.238279 19419 tablet_service.cc:1467] Processing CreateTablet for tablet 3890388daa25434993b6d6bbed1f0fd0 (DEFAULT_TABLE table=test-table [id=672c70e29c704ff6a2671cbbf2e10089]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:15:56.238693 19419 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3890388daa25434993b6d6bbed1f0fd0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:15:56.240885 19612 tablet_bootstrap.cc:492] T 3890388daa25434993b6d6bbed1f0fd0 P ce1bf2b4620340b88840348ea13eb73d: Bootstrap starting.
I20260620 20:15:56.241636 19614 tablet_bootstrap.cc:492] T 3890388daa25434993b6d6bbed1f0fd0 P 426541981bf84ca1a3e874674a81d687: Bootstrap starting.
I20260620 20:15:56.242506 19612 tablet_bootstrap.cc:654] T 3890388daa25434993b6d6bbed1f0fd0 P ce1bf2b4620340b88840348ea13eb73d: Neither blocks nor log segments found. Creating new log.
I20260620 20:15:56.242934 19614 tablet_bootstrap.cc:654] T 3890388daa25434993b6d6bbed1f0fd0 P 426541981bf84ca1a3e874674a81d687: Neither blocks nor log segments found. Creating new log.
I20260620 20:15:56.243839 19613 tablet_bootstrap.cc:492] T 3890388daa25434993b6d6bbed1f0fd0 P 56cd29ec76f24959a145a593ab44b2d0: Bootstrap starting.
I20260620 20:15:56.244941 19612 tablet_bootstrap.cc:492] T 3890388daa25434993b6d6bbed1f0fd0 P ce1bf2b4620340b88840348ea13eb73d: No bootstrap required, opened a new log
I20260620 20:15:56.245114 19612 ts_tablet_manager.cc:1397] T 3890388daa25434993b6d6bbed1f0fd0 P ce1bf2b4620340b88840348ea13eb73d: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260620 20:15:56.245656 19613 tablet_bootstrap.cc:654] T 3890388daa25434993b6d6bbed1f0fd0 P 56cd29ec76f24959a145a593ab44b2d0: Neither blocks nor log segments found. Creating new log.
I20260620 20:15:56.245785 19614 tablet_bootstrap.cc:492] T 3890388daa25434993b6d6bbed1f0fd0 P 426541981bf84ca1a3e874674a81d687: No bootstrap required, opened a new log
I20260620 20:15:56.245895 19612 raft_consensus.cc:348] T 3890388daa25434993b6d6bbed1f0fd0 P ce1bf2b4620340b88840348ea13eb73d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "426541981bf84ca1a3e874674a81d687" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 40783 } } peers { permanent_uuid: "ce1bf2b4620340b88840348ea13eb73d" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 40071 } } peers { permanent_uuid: "56cd29ec76f24959a145a593ab44b2d0" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 45785 } }
I20260620 20:15:56.246143 19612 raft_consensus.cc:374] T 3890388daa25434993b6d6bbed1f0fd0 P ce1bf2b4620340b88840348ea13eb73d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:15:56.245970 19614 ts_tablet_manager.cc:1397] T 3890388daa25434993b6d6bbed1f0fd0 P 426541981bf84ca1a3e874674a81d687: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260620 20:15:56.246225 19612 raft_consensus.cc:729] T 3890388daa25434993b6d6bbed1f0fd0 P ce1bf2b4620340b88840348ea13eb73d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce1bf2b4620340b88840348ea13eb73d, State: Initialized, Role: FOLLOWER
I20260620 20:15:56.246459 19612 consensus_queue.cc:260] T 3890388daa25434993b6d6bbed1f0fd0 P ce1bf2b4620340b88840348ea13eb73d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "426541981bf84ca1a3e874674a81d687" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 40783 } } peers { permanent_uuid: "ce1bf2b4620340b88840348ea13eb73d" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 40071 } } peers { permanent_uuid: "56cd29ec76f24959a145a593ab44b2d0" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 45785 } }
I20260620 20:15:56.246986 19612 ts_tablet_manager.cc:1428] T 3890388daa25434993b6d6bbed1f0fd0 P ce1bf2b4620340b88840348ea13eb73d: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260620 20:15:56.246924 19614 raft_consensus.cc:348] T 3890388daa25434993b6d6bbed1f0fd0 P 426541981bf84ca1a3e874674a81d687 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "426541981bf84ca1a3e874674a81d687" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 40783 } } peers { permanent_uuid: "ce1bf2b4620340b88840348ea13eb73d" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 40071 } } peers { permanent_uuid: "56cd29ec76f24959a145a593ab44b2d0" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 45785 } }
I20260620 20:15:56.247175 19614 raft_consensus.cc:374] T 3890388daa25434993b6d6bbed1f0fd0 P 426541981bf84ca1a3e874674a81d687 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:15:56.247246 19614 raft_consensus.cc:729] T 3890388daa25434993b6d6bbed1f0fd0 P 426541981bf84ca1a3e874674a81d687 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 426541981bf84ca1a3e874674a81d687, State: Initialized, Role: FOLLOWER
I20260620 20:15:56.247473 19614 consensus_queue.cc:260] T 3890388daa25434993b6d6bbed1f0fd0 P 426541981bf84ca1a3e874674a81d687 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "426541981bf84ca1a3e874674a81d687" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 40783 } } peers { permanent_uuid: "ce1bf2b4620340b88840348ea13eb73d" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 40071 } } peers { permanent_uuid: "56cd29ec76f24959a145a593ab44b2d0" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 45785 } }
I20260620 20:15:56.247998 19614 ts_tablet_manager.cc:1428] T 3890388daa25434993b6d6bbed1f0fd0 P 426541981bf84ca1a3e874674a81d687: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260620 20:15:56.248787 19613 tablet_bootstrap.cc:492] T 3890388daa25434993b6d6bbed1f0fd0 P 56cd29ec76f24959a145a593ab44b2d0: No bootstrap required, opened a new log
I20260620 20:15:56.248960 19613 ts_tablet_manager.cc:1397] T 3890388daa25434993b6d6bbed1f0fd0 P 56cd29ec76f24959a145a593ab44b2d0: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20260620 20:15:56.249665 19613 raft_consensus.cc:348] T 3890388daa25434993b6d6bbed1f0fd0 P 56cd29ec76f24959a145a593ab44b2d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "426541981bf84ca1a3e874674a81d687" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 40783 } } peers { permanent_uuid: "ce1bf2b4620340b88840348ea13eb73d" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 40071 } } peers { permanent_uuid: "56cd29ec76f24959a145a593ab44b2d0" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 45785 } }
I20260620 20:15:56.250458 19613 raft_consensus.cc:374] T 3890388daa25434993b6d6bbed1f0fd0 P 56cd29ec76f24959a145a593ab44b2d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:15:56.250592 19613 raft_consensus.cc:729] T 3890388daa25434993b6d6bbed1f0fd0 P 56cd29ec76f24959a145a593ab44b2d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56cd29ec76f24959a145a593ab44b2d0, State: Initialized, Role: FOLLOWER
I20260620 20:15:56.250864 19613 consensus_queue.cc:260] T 3890388daa25434993b6d6bbed1f0fd0 P 56cd29ec76f24959a145a593ab44b2d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "426541981bf84ca1a3e874674a81d687" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 40783 } } peers { permanent_uuid: "ce1bf2b4620340b88840348ea13eb73d" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 40071 } } peers { permanent_uuid: "56cd29ec76f24959a145a593ab44b2d0" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 45785 } }
I20260620 20:15:56.251459 19613 ts_tablet_manager.cc:1428] T 3890388daa25434993b6d6bbed1f0fd0 P 56cd29ec76f24959a145a593ab44b2d0: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260620 20:15:56.392441 19626 raft_consensus.cc:482] T 3890388daa25434993b6d6bbed1f0fd0 P ce1bf2b4620340b88840348ea13eb73d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 20:15:56.392639 19626 raft_consensus.cc:504] T 3890388daa25434993b6d6bbed1f0fd0 P ce1bf2b4620340b88840348ea13eb73d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "426541981bf84ca1a3e874674a81d687" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 40783 } } peers { permanent_uuid: "ce1bf2b4620340b88840348ea13eb73d" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 40071 } } peers { permanent_uuid: "56cd29ec76f24959a145a593ab44b2d0" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 45785 } }
I20260620 20:15:56.393370 19626 leader_election.cc:288] T 3890388daa25434993b6d6bbed1f0fd0 P ce1bf2b4620340b88840348ea13eb73d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 426541981bf84ca1a3e874674a81d687 (127.16.217.1:40783), 56cd29ec76f24959a145a593ab44b2d0 (127.16.217.3:45785)
I20260620 20:15:56.394021 19444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3890388daa25434993b6d6bbed1f0fd0" candidate_uuid: "ce1bf2b4620340b88840348ea13eb73d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56cd29ec76f24959a145a593ab44b2d0" is_pre_election: true
I20260620 20:15:56.394045 19162 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3890388daa25434993b6d6bbed1f0fd0" candidate_uuid: "ce1bf2b4620340b88840348ea13eb73d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "426541981bf84ca1a3e874674a81d687" is_pre_election: true
I20260620 20:15:56.394255 19444 raft_consensus.cc:2454] T 3890388daa25434993b6d6bbed1f0fd0 P 56cd29ec76f24959a145a593ab44b2d0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce1bf2b4620340b88840348ea13eb73d in term 0.
I20260620 20:15:56.394295 19162 raft_consensus.cc:2454] T 3890388daa25434993b6d6bbed1f0fd0 P 426541981bf84ca1a3e874674a81d687 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce1bf2b4620340b88840348ea13eb73d in term 0.
I20260620 20:15:56.394752 19235 leader_election.cc:302] T 3890388daa25434993b6d6bbed1f0fd0 P ce1bf2b4620340b88840348ea13eb73d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 56cd29ec76f24959a145a593ab44b2d0, ce1bf2b4620340b88840348ea13eb73d; no voters:
I20260620 20:15:56.395037 19626 raft_consensus.cc:2781] T 3890388daa25434993b6d6bbed1f0fd0 P ce1bf2b4620340b88840348ea13eb73d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 20:15:56.395128 19626 raft_consensus.cc:482] T 3890388daa25434993b6d6bbed1f0fd0 P ce1bf2b4620340b88840348ea13eb73d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 20:15:56.395265 19626 raft_consensus.cc:3037] T 3890388daa25434993b6d6bbed1f0fd0 P ce1bf2b4620340b88840348ea13eb73d [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:15:56.397298 19626 raft_consensus.cc:504] T 3890388daa25434993b6d6bbed1f0fd0 P ce1bf2b4620340b88840348ea13eb73d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "426541981bf84ca1a3e874674a81d687" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 40783 } } peers { permanent_uuid: "ce1bf2b4620340b88840348ea13eb73d" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 40071 } } peers { permanent_uuid: "56cd29ec76f24959a145a593ab44b2d0" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 45785 } }
I20260620 20:15:56.397887 19626 leader_election.cc:288] T 3890388daa25434993b6d6bbed1f0fd0 P ce1bf2b4620340b88840348ea13eb73d [CANDIDATE]: Term 1 election: Requested vote from peers 426541981bf84ca1a3e874674a81d687 (127.16.217.1:40783), 56cd29ec76f24959a145a593ab44b2d0 (127.16.217.3:45785)
I20260620 20:15:56.398365 19444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3890388daa25434993b6d6bbed1f0fd0" candidate_uuid: "ce1bf2b4620340b88840348ea13eb73d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56cd29ec76f24959a145a593ab44b2d0"
I20260620 20:15:56.398372 19162 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3890388daa25434993b6d6bbed1f0fd0" candidate_uuid: "ce1bf2b4620340b88840348ea13eb73d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "426541981bf84ca1a3e874674a81d687"
I20260620 20:15:56.398572 19162 raft_consensus.cc:3037] T 3890388daa25434993b6d6bbed1f0fd0 P 426541981bf84ca1a3e874674a81d687 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:15:56.398571 19444 raft_consensus.cc:3037] T 3890388daa25434993b6d6bbed1f0fd0 P 56cd29ec76f24959a145a593ab44b2d0 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:15:56.401441 19162 raft_consensus.cc:2454] T 3890388daa25434993b6d6bbed1f0fd0 P 426541981bf84ca1a3e874674a81d687 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce1bf2b4620340b88840348ea13eb73d in term 1.
I20260620 20:15:56.401477 19444 raft_consensus.cc:2454] T 3890388daa25434993b6d6bbed1f0fd0 P 56cd29ec76f24959a145a593ab44b2d0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce1bf2b4620340b88840348ea13eb73d in term 1.
I20260620 20:15:56.401919 19232 leader_election.cc:302] T 3890388daa25434993b6d6bbed1f0fd0 P ce1bf2b4620340b88840348ea13eb73d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 426541981bf84ca1a3e874674a81d687, ce1bf2b4620340b88840348ea13eb73d; no voters:
I20260620 20:15:56.402235 19626 raft_consensus.cc:2781] T 3890388daa25434993b6d6bbed1f0fd0 P ce1bf2b4620340b88840348ea13eb73d [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:15:56.402396 19626 raft_consensus.cc:686] T 3890388daa25434993b6d6bbed1f0fd0 P ce1bf2b4620340b88840348ea13eb73d [term 1 LEADER]: Becoming Leader. State: Replica: ce1bf2b4620340b88840348ea13eb73d, State: Running, Role: LEADER
I20260620 20:15:56.402669 19626 consensus_queue.cc:237] T 3890388daa25434993b6d6bbed1f0fd0 P ce1bf2b4620340b88840348ea13eb73d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "426541981bf84ca1a3e874674a81d687" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 40783 } } peers { permanent_uuid: "ce1bf2b4620340b88840348ea13eb73d" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 40071 } } peers { permanent_uuid: "56cd29ec76f24959a145a593ab44b2d0" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 45785 } }
I20260620 20:15:56.405157 19004 catalog_manager.cc:5495] T 3890388daa25434993b6d6bbed1f0fd0 P ce1bf2b4620340b88840348ea13eb73d reported cstate change: term changed from 0 to 1, leader changed from <none> to ce1bf2b4620340b88840348ea13eb73d (127.16.217.2). New cstate: current_term: 1 leader_uuid: "ce1bf2b4620340b88840348ea13eb73d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "426541981bf84ca1a3e874674a81d687" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 40783 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce1bf2b4620340b88840348ea13eb73d" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 40071 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "56cd29ec76f24959a145a593ab44b2d0" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 45785 } health_report { overall_health: UNKNOWN } } }
I20260620 20:15:56.459151 19162 raft_consensus.cc:1261] T 3890388daa25434993b6d6bbed1f0fd0 P 426541981bf84ca1a3e874674a81d687 [term 1 FOLLOWER]: Refusing update from remote peer ce1bf2b4620340b88840348ea13eb73d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:15:56.459324 19444 raft_consensus.cc:1261] T 3890388daa25434993b6d6bbed1f0fd0 P 56cd29ec76f24959a145a593ab44b2d0 [term 1 FOLLOWER]: Refusing update from remote peer ce1bf2b4620340b88840348ea13eb73d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:15:56.459751 19626 consensus_queue.cc:1035] T 3890388daa25434993b6d6bbed1f0fd0 P ce1bf2b4620340b88840348ea13eb73d [LEADER]: Connected to new peer: Peer: permanent_uuid: "426541981bf84ca1a3e874674a81d687" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 40783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:15:56.460050 19579 consensus_queue.cc:1035] T 3890388daa25434993b6d6bbed1f0fd0 P ce1bf2b4620340b88840348ea13eb73d [LEADER]: Connected to new peer: Peer: permanent_uuid: "56cd29ec76f24959a145a593ab44b2d0" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 45785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 20:15:56.480410 19585 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260620 20:15:58.076062 19003 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55292:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 20:15:58.076268 19003 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55292:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 20:15:58.080448 19003 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ebfe81c8d02342e6bc65b45342cd2b9d: Sending DeleteTablet for 3 replicas of tablet 3890388daa25434993b6d6bbed1f0fd0
I20260620 20:15:58.081321 19281 tablet_service.cc:1514] Processing DeleteTablet for tablet 3890388daa25434993b6d6bbed1f0fd0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:15:58 UTC) from {username='kudu', principal='kudu/127.16.217.62@KRBTEST.COM'} at 127.0.0.1:45486
I20260620 20:15:58.081353 19419 tablet_service.cc:1514] Processing DeleteTablet for tablet 3890388daa25434993b6d6bbed1f0fd0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:15:58 UTC) from {username='kudu', principal='kudu/127.16.217.62@KRBTEST.COM'} at 127.0.0.1:41296
I20260620 20:15:58.081373 19141 tablet_service.cc:1514] Processing DeleteTablet for tablet 3890388daa25434993b6d6bbed1f0fd0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:15:58 UTC) from {username='kudu', principal='kudu/127.16.217.62@KRBTEST.COM'} at 127.0.0.1:38800
I20260620 20:15:58.082499 17252 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu pid 19075
I20260620 20:15:58.085260 19656 tablet_replica.cc:330] T 3890388daa25434993b6d6bbed1f0fd0 P 426541981bf84ca1a3e874674a81d687: stopping tablet replica
I20260620 20:15:58.085474 19655 tablet_replica.cc:330] T 3890388daa25434993b6d6bbed1f0fd0 P ce1bf2b4620340b88840348ea13eb73d: stopping tablet replica
I20260620 20:15:58.085610 19656 raft_consensus.cc:2229] T 3890388daa25434993b6d6bbed1f0fd0 P 426541981bf84ca1a3e874674a81d687 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 20:15:58.085805 19655 raft_consensus.cc:2229] T 3890388daa25434993b6d6bbed1f0fd0 P ce1bf2b4620340b88840348ea13eb73d [term 1 LEADER]: Raft consensus shutting down.
I20260620 20:15:58.085986 19656 raft_consensus.cc:2258] T 3890388daa25434993b6d6bbed1f0fd0 P 426541981bf84ca1a3e874674a81d687 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:15:58.086326 19655 raft_consensus.cc:2258] T 3890388daa25434993b6d6bbed1f0fd0 P ce1bf2b4620340b88840348ea13eb73d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:15:58.087097 19654 tablet_replica.cc:330] T 3890388daa25434993b6d6bbed1f0fd0 P 56cd29ec76f24959a145a593ab44b2d0: stopping tablet replica
I20260620 20:15:58.087311 19656 ts_tablet_manager.cc:1905] T 3890388daa25434993b6d6bbed1f0fd0 P 426541981bf84ca1a3e874674a81d687: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:15:58.087387 19654 raft_consensus.cc:2229] T 3890388daa25434993b6d6bbed1f0fd0 P 56cd29ec76f24959a145a593ab44b2d0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 20:15:58.087785 19654 raft_consensus.cc:2258] T 3890388daa25434993b6d6bbed1f0fd0 P 56cd29ec76f24959a145a593ab44b2d0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:15:58.088001 19655 ts_tablet_manager.cc:1905] T 3890388daa25434993b6d6bbed1f0fd0 P ce1bf2b4620340b88840348ea13eb73d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:15:58.089288 19654 ts_tablet_manager.cc:1905] T 3890388daa25434993b6d6bbed1f0fd0 P 56cd29ec76f24959a145a593ab44b2d0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:15:58.094633 19655 ts_tablet_manager.cc:1918] T 3890388daa25434993b6d6bbed1f0fd0 P ce1bf2b4620340b88840348ea13eb73d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 20:15:58.094835 19655 log.cc:1196] T 3890388daa25434993b6d6bbed1f0fd0 P ce1bf2b4620340b88840348ea13eb73d: Deleting WAL directory at /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986536687152-17252-0/minicluster-data/ts-1/wal/wals/3890388daa25434993b6d6bbed1f0fd0
I20260620 20:15:58.095181 19656 ts_tablet_manager.cc:1918] T 3890388daa25434993b6d6bbed1f0fd0 P 426541981bf84ca1a3e874674a81d687: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 20:15:58.095358 19655 ts_tablet_manager.cc:1939] T 3890388daa25434993b6d6bbed1f0fd0 P ce1bf2b4620340b88840348ea13eb73d: Deleting consensus metadata
I20260620 20:15:58.095414 19656 log.cc:1196] T 3890388daa25434993b6d6bbed1f0fd0 P 426541981bf84ca1a3e874674a81d687: Deleting WAL directory at /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986536687152-17252-0/minicluster-data/ts-0/wal/wals/3890388daa25434993b6d6bbed1f0fd0
I20260620 20:15:58.095993 19656 ts_tablet_manager.cc:1939] T 3890388daa25434993b6d6bbed1f0fd0 P 426541981bf84ca1a3e874674a81d687: Deleting consensus metadata
Jun 20 20:15:58 dist-test-slave-2hpg krb5kdc[18959](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986555, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.16.217.1@KRBTEST.COM
I20260620 20:15:58.096648 18988 catalog_manager.cc:4841] TS ce1bf2b4620340b88840348ea13eb73d (127.16.217.2:40071): tablet 3890388daa25434993b6d6bbed1f0fd0 (table test-table [id=672c70e29c704ff6a2671cbbf2e10089]) successfully deleted
I20260620 20:15:58.096943 19654 ts_tablet_manager.cc:1918] T 3890388daa25434993b6d6bbed1f0fd0 P 56cd29ec76f24959a145a593ab44b2d0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 20:15:58.097118 19654 log.cc:1196] T 3890388daa25434993b6d6bbed1f0fd0 P 56cd29ec76f24959a145a593ab44b2d0: Deleting WAL directory at /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986536687152-17252-0/minicluster-data/ts-2/wal/wals/3890388daa25434993b6d6bbed1f0fd0
I20260620 20:15:58.097730 19654 ts_tablet_manager.cc:1939] T 3890388daa25434993b6d6bbed1f0fd0 P 56cd29ec76f24959a145a593ab44b2d0: Deleting consensus metadata
I20260620 20:15:58.097726 18988 catalog_manager.cc:4841] TS 426541981bf84ca1a3e874674a81d687 (127.16.217.1:40783): tablet 3890388daa25434993b6d6bbed1f0fd0 (table test-table [id=672c70e29c704ff6a2671cbbf2e10089]) successfully deleted
I20260620 20:15:58.098765 18991 catalog_manager.cc:4841] TS 56cd29ec76f24959a145a593ab44b2d0 (127.16.217.3:45785): tablet 3890388daa25434993b6d6bbed1f0fd0 (table test-table [id=672c70e29c704ff6a2671cbbf2e10089]) successfully deleted
I20260620 20:15:58.104027 19658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:58.083946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38834 (local address 127.16.217.1:40783)
0620 20:15:58.084773 (+ 827us) server_negotiation.cc:206] Beginning negotiation
0620 20:15:58.084783 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 20:15:58.085755 (+ 972us) server_negotiation.cc:407] Connection header received
0620 20:15:58.086122 (+ 367us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:15:58.086131 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:15:58.086311 (+ 180us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:15:58.086413 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:15:58.088823 (+ 2410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:58.089989 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:58.091735 (+ 1746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:58.092419 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:58.092523 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:15:58.097894 (+ 5371us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:15:58.097916 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:15:58.097945 (+ 29us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:15:58.097981 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:15:58.101000 (+ 3019us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:15:58.101589 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:15:58.101597 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:15:58.101599 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:15:58.101746 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:15:58.102208 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:15:58.102212 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:15:58.102214 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:15:58.102973 (+ 759us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:15:58.103219 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0620 20:15:58.103592 (+ 373us) server_negotiation.cc:299] Negotiation successful
0620 20:15:58.103761 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":601,"thread_start_us":445,"threads_started":1}
W20260620 20:15:58.104880 19207 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38834
W20260620 20:15:58.105588 17252 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu pid 19075: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 20:15:58.105774 17252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu with pid 19075
I20260620 20:15:58.134086 17252 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu pid 19216
Jun 20 20:15:58 dist-test-slave-2hpg krb5kdc[18959](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986555, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.16.217.2@KRBTEST.COM
I20260620 20:15:58.150399 19659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:58.134860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45522 (local address 127.16.217.2:40071)
0620 20:15:58.135394 (+ 534us) server_negotiation.cc:206] Beginning negotiation
0620 20:15:58.135399 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:15:58.135415 (+ 16us) server_negotiation.cc:407] Connection header received
0620 20:15:58.135467 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:15:58.135470 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:15:58.135527 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:15:58.135616 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:15:58.136847 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:58.137559 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:58.139305 (+ 1746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:58.139733 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:58.139830 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:15:58.144450 (+ 4620us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:15:58.144465 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:15:58.144481 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:15:58.144523 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:15:58.147443 (+ 2920us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:15:58.148312 (+ 869us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:15:58.148320 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:15:58.148322 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:15:58.148442 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:15:58.148798 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:15:58.148801 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:15:58.148803 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:15:58.149163 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:15:58.149384 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0620 20:15:58.149699 (+ 315us) server_negotiation.cc:299] Negotiation successful
0620 20:15:58.149968 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":430,"thread_start_us":294,"threads_started":1}
W20260620 20:15:58.151436 19346 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45522
W20260620 20:15:58.152087 17252 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu pid 19216: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 20:15:58.152217 17252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu with pid 19216
I20260620 20:15:58.175058 17252 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu pid 19355
Jun 20 20:15:58 dist-test-slave-2hpg krb5kdc[18959](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986555, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.16.217.3@KRBTEST.COM
I20260620 20:15:58.190706 19660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:58.175940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41310 (local address 127.16.217.3:45785)
0620 20:15:58.176527 (+ 587us) server_negotiation.cc:206] Beginning negotiation
0620 20:15:58.176532 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:15:58.176548 (+ 16us) server_negotiation.cc:407] Connection header received
0620 20:15:58.176599 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:15:58.176603 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:15:58.176658 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:15:58.176753 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:15:58.178077 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:58.178836 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:58.180026 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:58.180642 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:58.180768 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:15:58.184982 (+ 4214us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:15:58.184997 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:15:58.185012 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:15:58.185055 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:15:58.188126 (+ 3071us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:15:58.188749 (+ 623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:15:58.188757 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:15:58.188759 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:15:58.188926 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:15:58.189331 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:15:58.189335 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:15:58.189337 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:15:58.189688 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:15:58.190057 (+ 369us) server_negotiation.cc:1035] Waiting for connection context
0620 20:15:58.190170 (+ 113us) server_negotiation.cc:299] Negotiation successful
0620 20:15:58.190323 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":461,"thread_start_us":305,"threads_started":1}
W20260620 20:15:58.191375 19485 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41310
W20260620 20:15:58.191980 17252 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu pid 19355: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 20:15:58.192116 17252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu with pid 19355
I20260620 20:15:58.209277 17252 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu pid 18972
I20260620 20:15:58.210435 19044 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:15:58.326018 17252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu with pid 18972
2026-06-20T20:15:58Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (8743 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986536687152-17252-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 20:15:58 dist-test-slave-2hpg krb5kdc[19664](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 20:15:58 dist-test-slave-2hpg krb5kdc[19664](info): set up 2 sockets
Jun 20 20:15:58 dist-test-slave-2hpg krb5kdc[19664](info): commencing operation
krb5kdc: starting...
W20260620 20:15:59.448028 17252 mini_kdc.cc:121] Time spent starting KDC: real 1.078s user 0.002s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
W20260620 20:15:59.676266 17252 mini_kdc.cc:253] Time spent creating user principal test-admin: real 0.228s user 0.001s sys 0.005s
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
W20260620 20:15:59.878651 17252 mini_kdc.cc:253] Time spent creating user principal test-user: real 0.202s user 0.001s sys 0.005s
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
W20260620 20:16:00.217335 17252 mini_kdc.cc:253] Time spent creating user principal joe-interloper: real 0.338s user 0.001s sys 0.005s
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986536687152-17252-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986536687152-17252-0/krb5kdc/test-user.keytab.
W20260620 20:16:01.234886 17252 mini_kdc.cc:302] Time spent creating keytab for test-user: real 1.017s user 0.004s sys 0.000s
Jun 20 20:16:01 dist-test-slave-2hpg krb5kdc[19664](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986561, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T20:16:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T20:16:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.217.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.217.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.217.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.62.keytab.
Entry for principal kudu/127.16.217.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.62.keytab.
W20260620 20:16:02.189323 17252 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.16.217.62: real 0.904s user 0.004s sys 0.004s
I20260620 20:16:02.190907 17252 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
/tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986536687152-17252-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986536687152-17252-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986536687152-17252-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986536687152-17252-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.217.62:44007
--webserver_interface=127.16.217.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.62.keytab
--principal=kudu/127.16.217.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.217.20:36717
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.217.62:44007
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986536687152-17252-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986536687152-17252-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986536687152-17252-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:16:02.526927 19677 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:16:02.527210 19677 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:16:02.527297 19677 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:16:02.534610 19677 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:16:02.534711 19677 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:16:02.534756 19677 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:16:02.534813 19677 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:16:02.534858 19677 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:16:02.544631 19677 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.217.20:36717
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986536687152-17252-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986536687152-17252-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.217.62:44007
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.62.keytab
--principal=kudu/127.16.217.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.217.62:44007
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986536687152-17252-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.217.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986536687152-17252-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:16:02.545900 19677 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:16:02.547137 19677 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:16:02.559656 19682 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:16:02.559612 19683 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:16:02.559686 19685 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:16:02.560025 19677 server_base.cc:1034] running on GCE node
I20260620 20:16:02.561048 19677 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:16:02.562552 19677 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:16:02.563834 19677 hybrid_clock.cc:648] HybridClock initialized: now 1781986562563771 us; error 56 us; skew 500 ppm
I20260620 20:16:02.564218 19677 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:16:02 dist-test-slave-2hpg krb5kdc[19664](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986562, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.217.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:16:02.567229 19677 init.cc:377] Logged in from keytab as kudu/127.16.217.62@KRBTEST.COM (short username kudu)
I20260620 20:16:02.568864 19677 webserver.cc:460] Webserver started at http://127.16.217.62:32803/ using document root <none> and password file <none>
I20260620 20:16:02.569540 19677 fs_manager.cc:359] Metadata directory not provided
I20260620 20:16:02.569681 19677 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:16:02.570070 19677 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:16:02.573439 19677 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986536687152-17252-0/minicluster-data/master-0/data/instance:
uuid: "f0317562196b4a0cb5e21f8e1989283a"
format_stamp: "Formatted at 2026-06-20 20:16:02 on dist-test-slave-2hpg"
server_key: "95f47b75dc90c7bf3c4ba0268a5f6065"
server_key_iv: "7edee00362bf1473098dc74133213a63"
server_key_version: "encryptionkey@0"
I20260620 20:16:02.574229 19677 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986536687152-17252-0/minicluster-data/master-0/wal/instance:
uuid: "f0317562196b4a0cb5e21f8e1989283a"
format_stamp: "Formatted at 2026-06-20 20:16:02 on dist-test-slave-2hpg"
server_key: "95f47b75dc90c7bf3c4ba0268a5f6065"
server_key_iv: "7edee00362bf1473098dc74133213a63"
server_key_version: "encryptionkey@0"
I20260620 20:16:02.579209 19677 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260620 20:16:02.582746 19692 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:16:02.584125 19677 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 20:16:02.584280 19677 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986536687152-17252-0/minicluster-data/master-0/data,/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986536687152-17252-0/minicluster-data/master-0/wal
uuid: "f0317562196b4a0cb5e21f8e1989283a"
format_stamp: "Formatted at 2026-06-20 20:16:02 on dist-test-slave-2hpg"
server_key: "95f47b75dc90c7bf3c4ba0268a5f6065"
server_key_iv: "7edee00362bf1473098dc74133213a63"
server_key_version: "encryptionkey@0"
I20260620 20:16:02.584453 19677 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986536687152-17252-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986536687152-17252-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986536687152-17252-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:16:02.596110 19677 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:16:02.600092 19677 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:16:02.600333 19677 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:16:02.618958 19677 rpc_server.cc:307] RPC server started. Bound to: 127.16.217.62:44007
I20260620 20:16:02.619004 19744 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.217.62:44007 every 8 connection(s)
I20260620 20:16:02.620409 19677 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986536687152-17252-0/minicluster-data/master-0/data/info.pb
I20260620 20:16:02.623970 17252 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu as pid 19677
I20260620 20:16:02.624185 17252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986536687152-17252-0/minicluster-data/master-0/wal/instance
I20260620 20:16:02.624428 19745 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:16:02.624965 17252 external_mini_cluster.cc:1428] Setting key bfde515ff6baed9516618a0ca0754a4f
I20260620 20:16:02.640479 19745 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f0317562196b4a0cb5e21f8e1989283a: Bootstrap starting.
Jun 20 20:16:02 dist-test-slave-2hpg krb5kdc[19664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986561, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.217.62@KRBTEST.COM
I20260620 20:16:02.644750 19745 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f0317562196b4a0cb5e21f8e1989283a: Neither blocks nor log segments found. Creating new log.
I20260620 20:16:02.646219 19745 log.cc:824] T 00000000000000000000000000000000 P f0317562196b4a0cb5e21f8e1989283a: Log is configured to *not* fsync() on all Append() calls
I20260620 20:16:02.649727 19745 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f0317562196b4a0cb5e21f8e1989283a: No bootstrap required, opened a new log
I20260620 20:16:02.651037 19748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:02.627099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47234 (local address 127.16.217.62:44007)
0620 20:16:02.628079 (+ 980us) server_negotiation.cc:206] Beginning negotiation
0620 20:16:02.628091 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 20:16:02.628133 (+ 42us) server_negotiation.cc:407] Connection header received
0620 20:16:02.629953 (+ 1820us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:16:02.629975 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:16:02.630404 (+ 429us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:16:02.630734 (+ 330us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:16:02.632761 (+ 2027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:02.634214 (+ 1453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:02.636175 (+ 1961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:02.636926 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:02.637227 (+ 301us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:02.642295 (+ 5068us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:16:02.642309 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:16:02.642330 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:16:02.642381 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:16:02.646490 (+ 4109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:16:02.647399 (+ 909us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:16:02.647407 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:16:02.647413 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:16:02.647584 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:16:02.647977 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:16:02.647981 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:16:02.647984 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:16:02.648642 (+ 658us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:16:02.648960 (+ 318us) server_negotiation.cc:1035] Waiting for connection context
0620 20:16:02.649228 (+ 268us) server_negotiation.cc:299] Negotiation successful
0620 20:16:02.649643 (+ 415us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":783,"thread_start_us":528,"threads_started":1}
I20260620 20:16:02.657423 19745 raft_consensus.cc:348] T 00000000000000000000000000000000 P f0317562196b4a0cb5e21f8e1989283a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0317562196b4a0cb5e21f8e1989283a" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 44007 } }
I20260620 20:16:02.657732 19745 raft_consensus.cc:374] T 00000000000000000000000000000000 P f0317562196b4a0cb5e21f8e1989283a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:16:02.657812 19745 raft_consensus.cc:729] T 00000000000000000000000000000000 P f0317562196b4a0cb5e21f8e1989283a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0317562196b4a0cb5e21f8e1989283a, State: Initialized, Role: FOLLOWER
I20260620 20:16:02.658357 19745 consensus_queue.cc:260] T 00000000000000000000000000000000 P f0317562196b4a0cb5e21f8e1989283a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0317562196b4a0cb5e21f8e1989283a" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 44007 } }
I20260620 20:16:02.658552 19745 raft_consensus.cc:388] T 00000000000000000000000000000000 P f0317562196b4a0cb5e21f8e1989283a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 20:16:02.658643 19745 raft_consensus.cc:482] T 00000000000000000000000000000000 P f0317562196b4a0cb5e21f8e1989283a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 20:16:02.658778 19745 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f0317562196b4a0cb5e21f8e1989283a [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:16:02.678880 19745 raft_consensus.cc:504] T 00000000000000000000000000000000 P f0317562196b4a0cb5e21f8e1989283a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0317562196b4a0cb5e21f8e1989283a" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 44007 } }
I20260620 20:16:02.679586 19745 leader_election.cc:302] T 00000000000000000000000000000000 P f0317562196b4a0cb5e21f8e1989283a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f0317562196b4a0cb5e21f8e1989283a; no voters:
I20260620 20:16:02.680325 19745 leader_election.cc:288] T 00000000000000000000000000000000 P f0317562196b4a0cb5e21f8e1989283a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 20:16:02.680456 19750 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f0317562196b4a0cb5e21f8e1989283a [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:16:02.683547 19750 raft_consensus.cc:686] T 00000000000000000000000000000000 P f0317562196b4a0cb5e21f8e1989283a [term 1 LEADER]: Becoming Leader. State: Replica: f0317562196b4a0cb5e21f8e1989283a, State: Running, Role: LEADER
I20260620 20:16:02.684155 19745 sys_catalog.cc:564] T 00000000000000000000000000000000 P f0317562196b4a0cb5e21f8e1989283a [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:16:02.684181 19750 consensus_queue.cc:237] T 00000000000000000000000000000000 P f0317562196b4a0cb5e21f8e1989283a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0317562196b4a0cb5e21f8e1989283a" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 44007 } }
I20260620 20:16:02.687265 19752 sys_catalog.cc:455] T 00000000000000000000000000000000 P f0317562196b4a0cb5e21f8e1989283a [sys.catalog]: SysCatalogTable state changed. Reason: New leader f0317562196b4a0cb5e21f8e1989283a. Latest consensus state: current_term: 1 leader_uuid: "f0317562196b4a0cb5e21f8e1989283a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0317562196b4a0cb5e21f8e1989283a" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 44007 } } }
I20260620 20:16:02.687242 19751 sys_catalog.cc:455] T 00000000000000000000000000000000 P f0317562196b4a0cb5e21f8e1989283a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f0317562196b4a0cb5e21f8e1989283a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0317562196b4a0cb5e21f8e1989283a" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 44007 } } }
I20260620 20:16:02.688413 19752 sys_catalog.cc:458] T 00000000000000000000000000000000 P f0317562196b4a0cb5e21f8e1989283a [sys.catalog]: This master's current role is: LEADER
I20260620 20:16:02.688427 19751 sys_catalog.cc:458] T 00000000000000000000000000000000 P f0317562196b4a0cb5e21f8e1989283a [sys.catalog]: This master's current role is: LEADER
I20260620 20:16:02.690006 19761 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 20:16:02.696163 19761 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 20:16:02.704752 19761 catalog_manager.cc:1347] Generated new cluster ID: b755d97c464544ffb612529a8e61f1ba
I20260620 20:16:02.704866 19761 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 20:16:02.713410 19761 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 20:16:02.714708 19761 catalog_manager.cc:1504] Loading token signing keys...
I20260620 20:16:02.724869 19761 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f0317562196b4a0cb5e21f8e1989283a: Generated new TSK 0
I20260620 20:16:02.725648 19761 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.217.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.217.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.217.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.1.keytab.
Entry for principal kudu/127.16.217.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.1.keytab.
W20260620 20:16:03.937820 17252 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.16.217.1: real 1.204s user 0.004s sys 0.005s
I20260620 20:16:03.938868 17252 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
/tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986536687152-17252-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986536687152-17252-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986536687152-17252-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986536687152-17252-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.217.1:0
--local_ip_for_outbound_sockets=127.16.217.1
--webserver_interface=127.16.217.1
--webserver_port=0
--tserver_master_addrs=127.16.217.62:44007
--keytab_file=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.1.keytab
--principal=kudu/127.16.217.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.217.20:36717
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986536687152-17252-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986536687152-17252-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986536687152-17252-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:16:04.258857 19771 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:16:04.259111 19771 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:16:04.259265 19771 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:16:04.266367 19771 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:16:04.266474 19771 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:16:04.266670 19771 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.217.1
I20260620 20:16:04.276182 19771 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.217.20:36717
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986536687152-17252-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986536687152-17252-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.1.keytab
--principal=kudu/127.16.217.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.217.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986536687152-17252-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.217.1
--webserver_port=0
--tserver_master_addrs=127.16.217.62:44007
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.217.1
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986536687152-17252-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:16:04.277438 19771 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:16:04.278749 19771 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:16:04.292034 19777 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:16:04.291994 19780 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:16:04.292063 19778 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:16:04.293037 19771 server_base.cc:1034] running on GCE node
I20260620 20:16:04.293613 19771 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:16:04.294621 19771 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:16:04.295838 19771 hybrid_clock.cc:648] HybridClock initialized: now 1781986564295779 us; error 55 us; skew 500 ppm
I20260620 20:16:04.296172 19771 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:16:04 dist-test-slave-2hpg krb5kdc[19664](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986564, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.217.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:16:04.299346 19771 init.cc:377] Logged in from keytab as kudu/127.16.217.1@KRBTEST.COM (short username kudu)
I20260620 20:16:04.300776 19771 webserver.cc:460] Webserver started at http://127.16.217.1:34837/ using document root <none> and password file <none>
I20260620 20:16:04.301477 19771 fs_manager.cc:359] Metadata directory not provided
I20260620 20:16:04.301635 19771 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:16:04.302004 19771 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:16:04.304845 19771 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986536687152-17252-0/minicluster-data/ts-0/data/instance:
uuid: "8c338223936a4264b3b7fc617378d777"
format_stamp: "Formatted at 2026-06-20 20:16:04 on dist-test-slave-2hpg"
server_key: "78e605b181c006ab53194af3b0861b53"
server_key_iv: "f3e93665b0c7ea22429a87d2e1aa2e0c"
server_key_version: "encryptionkey@0"
I20260620 20:16:04.305657 19771 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986536687152-17252-0/minicluster-data/ts-0/wal/instance:
uuid: "8c338223936a4264b3b7fc617378d777"
format_stamp: "Formatted at 2026-06-20 20:16:04 on dist-test-slave-2hpg"
server_key: "78e605b181c006ab53194af3b0861b53"
server_key_iv: "f3e93665b0c7ea22429a87d2e1aa2e0c"
server_key_version: "encryptionkey@0"
I20260620 20:16:04.310761 19771 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260620 20:16:04.314119 19787 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:16:04.315433 19771 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 20:16:04.315609 19771 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986536687152-17252-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986536687152-17252-0/minicluster-data/ts-0/wal
uuid: "8c338223936a4264b3b7fc617378d777"
format_stamp: "Formatted at 2026-06-20 20:16:04 on dist-test-slave-2hpg"
server_key: "78e605b181c006ab53194af3b0861b53"
server_key_iv: "f3e93665b0c7ea22429a87d2e1aa2e0c"
server_key_version: "encryptionkey@0"
I20260620 20:16:04.315802 19771 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986536687152-17252-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986536687152-17252-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986536687152-17252-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:16:04.339282 19771 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:16:04.343309 19771 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:16:04.343603 19771 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:16:04.344552 19771 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:16:04.345839 19771 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:16:04.345999 19771 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:16:04.346145 19771 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:16:04.346218 19771 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:16:04.379047 19771 rpc_server.cc:307] RPC server started. Bound to: 127.16.217.1:42849
I20260620 20:16:04.379096 19899 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.217.1:42849 every 8 connection(s)
I20260620 20:16:04.380311 19771 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986536687152-17252-0/minicluster-data/ts-0/data/info.pb
I20260620 20:16:04.387454 17252 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu as pid 19771
I20260620 20:16:04.387673 17252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986536687152-17252-0/minicluster-data/ts-0/wal/instance
I20260620 20:16:04.388463 17252 external_mini_cluster.cc:1428] Setting key 52cc2f9babea2c81793360d99aac3179
Jun 20 20:16:04 dist-test-slave-2hpg krb5kdc[19664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986564, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.217.1@KRBTEST.COM for kudu/127.16.217.62@KRBTEST.COM
I20260620 20:16:04.400375 19902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:04.382826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.217.1:42545 (local address 127.16.217.62:44007)
0620 20:16:04.383298 (+ 472us) server_negotiation.cc:206] Beginning negotiation
0620 20:16:04.383305 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:16:04.384184 (+ 879us) server_negotiation.cc:407] Connection header received
0620 20:16:04.386130 (+ 1946us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:16:04.386135 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:16:04.386209 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:16:04.386304 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:16:04.388080 (+ 1776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:04.389008 (+ 928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:04.390900 (+ 1892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:04.391289 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:04.391425 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:04.394859 (+ 3434us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:16:04.394870 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:16:04.394873 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:16:04.394903 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:16:04.397394 (+ 2491us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:16:04.398189 (+ 795us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:16:04.398198 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:16:04.398201 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:16:04.398315 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:16:04.398724 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:16:04.398728 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:16:04.398731 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:16:04.398986 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:16:04.399137 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0620 20:16:04.399791 (+ 654us) server_negotiation.cc:299] Negotiation successful
0620 20:16:04.400003 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":337,"thread_start_us":210,"threads_started":1}
I20260620 20:16:04.401335 19903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:04.383341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:44007 (local address 127.16.217.1:42545)
0620 20:16:04.384027 (+ 686us) negotiation.cc:107] Waiting for socket to connect
0620 20:16:04.384069 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 20:16:04.385837 (+ 1768us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:16:04.386512 (+ 675us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:16:04.386520 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:16:04.387031 (+ 511us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:16:04.387894 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:16:04.387921 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:16:04.389151 (+ 1230us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:16:04.389157 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:16:04.390761 (+ 1604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:16:04.390773 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:16:04.391520 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:16:04.391529 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:16:04.391781 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:04.392465 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:16:04.392486 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:16:04.394638 (+ 2152us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:16:04.397548 (+ 2910us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:16:04.397557 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:16:04.397575 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:16:04.398069 (+ 494us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:16:04.398424 (+ 355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:16:04.398430 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:16:04.398433 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:16:04.398609 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:16:04.399116 (+ 507us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:16:04.399124 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:16:04.399445 (+ 321us) client_negotiation.cc:769] Sending connection context
0620 20:16:04.399698 (+ 253us) client_negotiation.cc:240] Negotiation successful
0620 20:16:04.400234 (+ 536us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":254,"threads_started":1}
I20260620 20:16:04.402976 19900 heartbeater.cc:347] Connected to a master server at 127.16.217.62:44007
I20260620 20:16:04.403319 19900 heartbeater.cc:464] Registering TS with master...
I20260620 20:16:04.404194 19900 heartbeater.cc:511] Master 127.16.217.62:44007 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.16.217.2@KRBTEST.COM; defaulting to no policy
I20260620 20:16:04.406682 19710 ts_manager.cc:194] Registered new tserver with Master: 8c338223936a4264b3b7fc617378d777 (127.16.217.1:42849)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.217.2@KRBTEST.COM" created.
I20260620 20:16:04.408342 19710 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.217.1@KRBTEST.COM'} at 127.16.217.1:42545
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.217.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.2.keytab.
Entry for principal kudu/127.16.217.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.2.keytab.
I20260620 20:16:04.431145 17252 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
/tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986536687152-17252-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986536687152-17252-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986536687152-17252-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986536687152-17252-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.217.2:0
--local_ip_for_outbound_sockets=127.16.217.2
--webserver_interface=127.16.217.2
--webserver_port=0
--tserver_master_addrs=127.16.217.62:44007
--keytab_file=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.2.keytab
--principal=kudu/127.16.217.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.217.20:36717
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986536687152-17252-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986536687152-17252-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986536687152-17252-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:16:04.770045 19906 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:16:04.770309 19906 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:16:04.770402 19906 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:16:04.777400 19906 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:16:04.777503 19906 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:16:04.777619 19906 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.217.2
I20260620 20:16:04.787250 19906 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.217.20:36717
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986536687152-17252-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986536687152-17252-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.2.keytab
--principal=kudu/127.16.217.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.217.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986536687152-17252-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.217.2
--webserver_port=0
--tserver_master_addrs=127.16.217.62:44007
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.217.2
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986536687152-17252-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:16:04.788585 19906 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:16:04.790005 19906 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:16:04.803010 19912 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:16:04.803086 19914 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:16:04.803048 19911 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:16:04.804164 19906 server_base.cc:1034] running on GCE node
I20260620 20:16:04.804867 19906 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:16:04.805970 19906 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:16:04.807229 19906 hybrid_clock.cc:648] HybridClock initialized: now 1781986564807160 us; error 54 us; skew 500 ppm
I20260620 20:16:04.807564 19906 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:16:04 dist-test-slave-2hpg krb5kdc[19664](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986564, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.217.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:16:04.810933 19906 init.cc:377] Logged in from keytab as kudu/127.16.217.2@KRBTEST.COM (short username kudu)
I20260620 20:16:04.812667 19906 webserver.cc:460] Webserver started at http://127.16.217.2:40719/ using document root <none> and password file <none>
I20260620 20:16:04.813467 19906 fs_manager.cc:359] Metadata directory not provided
I20260620 20:16:04.813629 19906 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:16:04.814065 19906 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:16:04.817308 19906 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986536687152-17252-0/minicluster-data/ts-1/data/instance:
uuid: "41bb59bd0b2c48dcb69b2bbbf5dc01b9"
format_stamp: "Formatted at 2026-06-20 20:16:04 on dist-test-slave-2hpg"
server_key: "e04f19ee8e24b66412fd0863363ab612"
server_key_iv: "25f649de56650fc9b1f4f11621579fbc"
server_key_version: "encryptionkey@0"
I20260620 20:16:04.818279 19906 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986536687152-17252-0/minicluster-data/ts-1/wal/instance:
uuid: "41bb59bd0b2c48dcb69b2bbbf5dc01b9"
format_stamp: "Formatted at 2026-06-20 20:16:04 on dist-test-slave-2hpg"
server_key: "e04f19ee8e24b66412fd0863363ab612"
server_key_iv: "25f649de56650fc9b1f4f11621579fbc"
server_key_version: "encryptionkey@0"
I20260620 20:16:04.823664 19906 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 20:16:04.827205 19922 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:16:04.828519 19906 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 20:16:04.828680 19906 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986536687152-17252-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986536687152-17252-0/minicluster-data/ts-1/wal
uuid: "41bb59bd0b2c48dcb69b2bbbf5dc01b9"
format_stamp: "Formatted at 2026-06-20 20:16:04 on dist-test-slave-2hpg"
server_key: "e04f19ee8e24b66412fd0863363ab612"
server_key_iv: "25f649de56650fc9b1f4f11621579fbc"
server_key_version: "encryptionkey@0"
I20260620 20:16:04.828884 19906 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986536687152-17252-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986536687152-17252-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986536687152-17252-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:16:04.847874 19906 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:16:04.852025 19906 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:16:04.852311 19906 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:16:04.853235 19906 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:16:04.855014 19906 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:16:04.855109 19906 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:16:04.855247 19906 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:16:04.855355 19906 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:16:04.889128 19906 rpc_server.cc:307] RPC server started. Bound to: 127.16.217.2:36171
I20260620 20:16:04.889145 20034 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.217.2:36171 every 8 connection(s)
I20260620 20:16:04.890539 19906 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986536687152-17252-0/minicluster-data/ts-1/data/info.pb
I20260620 20:16:04.898351 17252 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu as pid 19906
I20260620 20:16:04.898564 17252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986536687152-17252-0/minicluster-data/ts-1/wal/instance
I20260620 20:16:04.899135 17252 external_mini_cluster.cc:1428] Setting key ca6533c4a40e9c4e38d722491c109c38
Jun 20 20:16:04 dist-test-slave-2hpg krb5kdc[19664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986564, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.217.2@KRBTEST.COM for kudu/127.16.217.62@KRBTEST.COM
I20260620 20:16:04.911049 19902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:04.893043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.217.2:36067 (local address 127.16.217.62:44007)
0620 20:16:04.893210 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0620 20:16:04.893214 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:16:04.894354 (+ 1140us) server_negotiation.cc:407] Connection header received
0620 20:16:04.895992 (+ 1638us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:16:04.895996 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:16:04.896060 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:16:04.896147 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:16:04.897521 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:04.898332 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:04.900281 (+ 1949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:04.900703 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:04.900837 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:04.905054 (+ 4217us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:16:04.905059 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:16:04.905063 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:16:04.905097 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:16:04.908103 (+ 3006us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:16:04.908815 (+ 712us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:16:04.908823 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:16:04.908828 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:16:04.908964 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:16:04.909383 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:16:04.909388 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:16:04.909392 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:16:04.909677 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:16:04.909828 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0620 20:16:04.910516 (+ 688us) server_negotiation.cc:299] Negotiation successful
0620 20:16:04.910723 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 20:16:04.912011 20037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:04.893451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:44007 (local address 127.16.217.2:36067)
0620 20:16:04.894178 (+ 727us) negotiation.cc:107] Waiting for socket to connect
0620 20:16:04.894220 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 20:16:04.895750 (+ 1530us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:16:04.896351 (+ 601us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:16:04.896358 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:16:04.896835 (+ 477us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:16:04.897365 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:16:04.897381 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:16:04.898481 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:16:04.898487 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:16:04.900125 (+ 1638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:16:04.900135 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:16:04.900923 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:16:04.900930 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:16:04.901225 (+ 295us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:04.902024 (+ 799us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:16:04.902067 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:16:04.904803 (+ 2736us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:16:04.908252 (+ 3449us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:16:04.908262 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:16:04.908280 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:16:04.908694 (+ 414us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:16:04.909073 (+ 379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:16:04.909080 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:16:04.909083 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:16:04.909267 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:16:04.909807 (+ 540us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:16:04.909815 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:16:04.910174 (+ 359us) client_negotiation.cc:769] Sending connection context
0620 20:16:04.910445 (+ 271us) client_negotiation.cc:240] Negotiation successful
0620 20:16:04.910958 (+ 513us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":516,"thread_start_us":265,"threads_started":1}
I20260620 20:16:04.913601 20035 heartbeater.cc:347] Connected to a master server at 127.16.217.62:44007
I20260620 20:16:04.913946 20035 heartbeater.cc:464] Registering TS with master...
I20260620 20:16:04.914636 20035 heartbeater.cc:511] Master 127.16.217.62:44007 requested a full tablet report, sending...
I20260620 20:16:04.916172 19710 ts_manager.cc:194] Registered new tserver with Master: 41bb59bd0b2c48dcb69b2bbbf5dc01b9 (127.16.217.2:36171)
WARNING: no policy specified for kudu/127.16.217.3@KRBTEST.COM; defaulting to no policy
I20260620 20:16:04.917400 19710 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.217.2@KRBTEST.COM'} at 127.16.217.2:36067
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.217.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.217.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.3.keytab.
Entry for principal kudu/127.16.217.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.3.keytab.
I20260620 20:16:04.943555 17252 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
/tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986536687152-17252-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986536687152-17252-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986536687152-17252-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986536687152-17252-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.217.3:0
--local_ip_for_outbound_sockets=127.16.217.3
--webserver_interface=127.16.217.3
--webserver_port=0
--tserver_master_addrs=127.16.217.62:44007
--keytab_file=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.3.keytab
--principal=kudu/127.16.217.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.217.20:36717
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986536687152-17252-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986536687152-17252-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986536687152-17252-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:16:05.267475 20040 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:16:05.267750 20040 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:16:05.267859 20040 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:16:05.275180 20040 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:16:05.275308 20040 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:16:05.275431 20040 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.217.3
I20260620 20:16:05.285225 20040 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.217.20:36717
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986536687152-17252-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986536687152-17252-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.3.keytab
--principal=kudu/127.16.217.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.217.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986536687152-17252-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.217.3
--webserver_port=0
--tserver_master_addrs=127.16.217.62:44007
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.217.3
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986536687152-17252-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:16:05.286545 20040 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:16:05.288060 20040 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:16:05.300834 20045 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:16:05.301057 20048 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:16:05.301075 20046 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:16:05.301947 20040 server_base.cc:1034] running on GCE node
I20260620 20:16:05.302584 20040 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:16:05.303510 20040 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:16:05.304785 20040 hybrid_clock.cc:648] HybridClock initialized: now 1781986565304704 us; error 60 us; skew 500 ppm
I20260620 20:16:05.305119 20040 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:16:05 dist-test-slave-2hpg krb5kdc[19664](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986565, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.217.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:16:05.308310 20040 init.cc:377] Logged in from keytab as kudu/127.16.217.3@KRBTEST.COM (short username kudu)
I20260620 20:16:05.309749 20040 webserver.cc:460] Webserver started at http://127.16.217.3:41505/ using document root <none> and password file <none>
I20260620 20:16:05.310530 20040 fs_manager.cc:359] Metadata directory not provided
I20260620 20:16:05.310670 20040 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:16:05.310971 20040 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:16:05.313815 20040 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986536687152-17252-0/minicluster-data/ts-2/data/instance:
uuid: "122f213182fc4e2fa540ea6242d9dcea"
format_stamp: "Formatted at 2026-06-20 20:16:05 on dist-test-slave-2hpg"
server_key: "e7de6327fd029c24c591d6c46d10873d"
server_key_iv: "9f06286039286ff3ca4c2d5e95a38667"
server_key_version: "encryptionkey@0"
I20260620 20:16:05.314706 20040 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986536687152-17252-0/minicluster-data/ts-2/wal/instance:
uuid: "122f213182fc4e2fa540ea6242d9dcea"
format_stamp: "Formatted at 2026-06-20 20:16:05 on dist-test-slave-2hpg"
server_key: "e7de6327fd029c24c591d6c46d10873d"
server_key_iv: "9f06286039286ff3ca4c2d5e95a38667"
server_key_version: "encryptionkey@0"
I20260620 20:16:05.320894 20040 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260620 20:16:05.324291 20055 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:16:05.325578 20040 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 20:16:05.325778 20040 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986536687152-17252-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986536687152-17252-0/minicluster-data/ts-2/wal
uuid: "122f213182fc4e2fa540ea6242d9dcea"
format_stamp: "Formatted at 2026-06-20 20:16:05 on dist-test-slave-2hpg"
server_key: "e7de6327fd029c24c591d6c46d10873d"
server_key_iv: "9f06286039286ff3ca4c2d5e95a38667"
server_key_version: "encryptionkey@0"
I20260620 20:16:05.326040 20040 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986536687152-17252-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986536687152-17252-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986536687152-17252-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:16:05.345589 20040 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:16:05.350965 20040 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:16:05.351372 20040 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:16:05.352911 20040 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:16:05.354840 20040 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:16:05.354944 20040 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:16:05.355080 20040 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:16:05.355149 20040 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:16:05.389629 20040 rpc_server.cc:307] RPC server started. Bound to: 127.16.217.3:33773
I20260620 20:16:05.389647 20168 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.217.3:33773 every 8 connection(s)
I20260620 20:16:05.390848 20040 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986536687152-17252-0/minicluster-data/ts-2/data/info.pb
I20260620 20:16:05.391207 17252 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu as pid 20040
I20260620 20:16:05.391340 17252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986536687152-17252-0/minicluster-data/ts-2/wal/instance
I20260620 20:16:05.391862 17252 external_mini_cluster.cc:1428] Setting key cdf4490dd728b60eefbbfcee473aad17
Jun 20 20:16:05 dist-test-slave-2hpg krb5kdc[19664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986565, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.217.3@KRBTEST.COM for kudu/127.16.217.62@KRBTEST.COM
I20260620 20:16:05.410285 19902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:05.393531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.217.3:33931 (local address 127.16.217.62:44007)
0620 20:16:05.393738 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0620 20:16:05.393751 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 20:16:05.395237 (+ 1486us) server_negotiation.cc:407] Connection header received
0620 20:16:05.396981 (+ 1744us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:16:05.396986 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:16:05.397073 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:16:05.397175 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:16:05.398613 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:05.399311 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:05.400829 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:05.401230 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:05.401362 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:05.404637 (+ 3275us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:16:05.404646 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:16:05.404649 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:16:05.404679 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:16:05.407713 (+ 3034us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:16:05.408413 (+ 700us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:16:05.408417 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:16:05.408419 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:16:05.408536 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:16:05.408918 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:16:05.408921 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:16:05.408923 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:16:05.409170 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:16:05.409270 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0620 20:16:05.409825 (+ 555us) server_negotiation.cc:299] Negotiation successful
0620 20:16:05.410059 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 20:16:05.411242 20171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:05.394206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:44007 (local address 127.16.217.3:33931)
0620 20:16:05.395070 (+ 864us) negotiation.cc:107] Waiting for socket to connect
0620 20:16:05.395143 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0620 20:16:05.396724 (+ 1581us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:16:05.397402 (+ 678us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:16:05.397410 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:16:05.397930 (+ 520us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:16:05.398460 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:16:05.398476 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:16:05.399465 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:16:05.399471 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:16:05.400683 (+ 1212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:16:05.400694 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:16:05.401443 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:16:05.401453 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:16:05.401693 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:05.402329 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:16:05.402353 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:16:05.404431 (+ 2078us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:16:05.407881 (+ 3450us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:16:05.407888 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:16:05.407899 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:16:05.408289 (+ 390us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:16:05.408626 (+ 337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:16:05.408629 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:16:05.408631 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:16:05.408787 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:16:05.409268 (+ 481us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:16:05.409273 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:16:05.409539 (+ 266us) client_negotiation.cc:769] Sending connection context
0620 20:16:05.409745 (+ 206us) client_negotiation.cc:240] Negotiation successful
0620 20:16:05.410194 (+ 449us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":499,"thread_start_us":286,"threads_started":1}
I20260620 20:16:05.411439 19900 heartbeater.cc:503] Master 127.16.217.62:44007 was elected leader, sending a full tablet report...
I20260620 20:16:05.413121 20169 heartbeater.cc:347] Connected to a master server at 127.16.217.62:44007
I20260620 20:16:05.413414 20169 heartbeater.cc:464] Registering TS with master...
I20260620 20:16:05.414153 20169 heartbeater.cc:511] Master 127.16.217.62:44007 requested a full tablet report, sending...
I20260620 20:16:05.415390 19709 ts_manager.cc:194] Registered new tserver with Master: 122f213182fc4e2fa540ea6242d9dcea (127.16.217.3:33773)
I20260620 20:16:05.416247 19709 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.217.3@KRBTEST.COM'} at 127.16.217.3:33931
I20260620 20:16:05.419301 17252 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260620 20:16:05.447744 19902 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:16:05.445134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47238 (local address 127.16.217.62:44007)
0620 20:16:05.445353 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0620 20:16:05.445361 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 20:16:05.445736 (+ 375us) server_negotiation.cc:407] Connection header received
0620 20:16:05.446035 (+ 299us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:16:05.446039 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:16:05.446104 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:16:05.446216 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:16:05.447470 (+ 1254us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47238: BlockingRecv error: recv got EOF from 127.0.0.1:47238 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 20:16:05.451387 17252 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu pid 19771
W20260620 20:16:05.454221 17252 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu pid 19771: Not authorized: Client connection negotiation failed: client connection to 127.16.217.1:42849: server requires authentication, but client does not have Kerberos credentials available
I20260620 20:16:05.454402 17252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu with pid 19771
I20260620 20:16:05.469776 17252 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu pid 19906
W20260620 20:16:05.472405 17252 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu pid 19906: Not authorized: Client connection negotiation failed: client connection to 127.16.217.2:36171: server requires authentication, but client does not have Kerberos credentials available
I20260620 20:16:05.472565 17252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu with pid 19906
I20260620 20:16:05.487867 17252 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu pid 20040
W20260620 20:16:05.490777 17252 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu pid 20040: Not authorized: Client connection negotiation failed: client connection to 127.16.217.3:33773: server requires authentication, but client does not have Kerberos credentials available
I20260620 20:16:05.490952 17252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu with pid 20040
I20260620 20:16:05.506165 17252 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu pid 19677
I20260620 20:16:05.507455 19740 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:16:05.608019 17252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu with pid 19677
2026-06-20T20:16:05Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (7278 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781986536687152-17252-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 20:16:05 dist-test-slave-2hpg krb5kdc[20186](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 20:16:05 dist-test-slave-2hpg krb5kdc[20186](info): set up 2 sockets
Jun 20 20:16:05 dist-test-slave-2hpg krb5kdc[20186](info): commencing operation
krb5kdc: starting...
W20260620 20:16:06.698442 17252 mini_kdc.cc:121] Time spent starting KDC: real 1.049s user 0.001s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781986536687152-17252-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781986536687152-17252-0/krb5kdc/test-user.keytab.
Jun 20 20:16:06 dist-test-slave-2hpg krb5kdc[20186](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986566, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T20:16:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T20:16:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.217.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.217.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.217.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.62.keytab.
Entry for principal kudu/127.16.217.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.62.keytab.
I20260620 20:16:06.885329 17252 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
/tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781986536687152-17252-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781986536687152-17252-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781986536687152-17252-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781986536687152-17252-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.217.62:32975
--webserver_interface=127.16.217.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.62.keytab
--principal=kudu/127.16.217.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.217.20:39301
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.217.62:32975
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781986536687152-17252-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781986536687152-17252-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781986536687152-17252-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:16:07.200166 20199 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:16:07.200579 20199 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:16:07.200740 20199 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:16:07.209527 20199 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:16:07.209640 20199 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:16:07.209753 20199 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:16:07.209844 20199 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:16:07.209926 20199 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:16:07.219753 20199 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.217.20:39301
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781986536687152-17252-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781986536687152-17252-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.217.62:32975
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.62.keytab
--principal=kudu/127.16.217.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.217.62:32975
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781986536687152-17252-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.217.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781986536687152-17252-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:16:07.221068 20199 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:16:07.222450 20199 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:16:07.234223 20204 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:16:07.234287 20205 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:16:07.234308 20207 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:16:07.235142 20199 server_base.cc:1034] running on GCE node
I20260620 20:16:07.235863 20199 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:16:07.237171 20199 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:16:07.238425 20199 hybrid_clock.cc:648] HybridClock initialized: now 1781986567238367 us; error 49 us; skew 500 ppm
I20260620 20:16:07.238794 20199 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:16:07 dist-test-slave-2hpg krb5kdc[20186](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986567, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.217.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:16:07.245366 20199 init.cc:377] Logged in from keytab as kudu/127.16.217.62@KRBTEST.COM (short username kudu)
I20260620 20:16:07.246863 20199 webserver.cc:460] Webserver started at http://127.16.217.62:39335/ using document root <none> and password file <none>
I20260620 20:16:07.247538 20199 fs_manager.cc:359] Metadata directory not provided
I20260620 20:16:07.247685 20199 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:16:07.247985 20199 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:16:07.250921 20199 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781986536687152-17252-0/minicluster-data/master-0/data/instance:
uuid: "92c04115f7a84793bb5b64eecd929e90"
format_stamp: "Formatted at 2026-06-20 20:16:07 on dist-test-slave-2hpg"
server_key: "7a5586e1a7d86f75f33dd5e5ddd781aa"
server_key_iv: "2f9241339afec0eb8074041aa734d05c"
server_key_version: "encryptionkey@0"
I20260620 20:16:07.251689 20199 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781986536687152-17252-0/minicluster-data/master-0/wal/instance:
uuid: "92c04115f7a84793bb5b64eecd929e90"
format_stamp: "Formatted at 2026-06-20 20:16:07 on dist-test-slave-2hpg"
server_key: "7a5586e1a7d86f75f33dd5e5ddd781aa"
server_key_iv: "2f9241339afec0eb8074041aa734d05c"
server_key_version: "encryptionkey@0"
I20260620 20:16:07.256613 20199 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 20:16:07.259828 20214 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:16:07.261060 20199 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 20:16:07.261229 20199 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781986536687152-17252-0/minicluster-data/master-0/data,/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781986536687152-17252-0/minicluster-data/master-0/wal
uuid: "92c04115f7a84793bb5b64eecd929e90"
format_stamp: "Formatted at 2026-06-20 20:16:07 on dist-test-slave-2hpg"
server_key: "7a5586e1a7d86f75f33dd5e5ddd781aa"
server_key_iv: "2f9241339afec0eb8074041aa734d05c"
server_key_version: "encryptionkey@0"
I20260620 20:16:07.261423 20199 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781986536687152-17252-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781986536687152-17252-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781986536687152-17252-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:16:07.294442 20199 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:16:07.298527 20199 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:16:07.298816 20199 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:16:07.316860 20199 rpc_server.cc:307] RPC server started. Bound to: 127.16.217.62:32975
I20260620 20:16:07.316895 20265 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.217.62:32975 every 8 connection(s)
I20260620 20:16:07.318260 20199 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781986536687152-17252-0/minicluster-data/master-0/data/info.pb
I20260620 20:16:07.321908 20266 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:16:07.324141 17252 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu as pid 20199
I20260620 20:16:07.324378 17252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781986536687152-17252-0/minicluster-data/master-0/wal/instance
I20260620 20:16:07.324967 17252 external_mini_cluster.cc:1428] Setting key 507faccb8df2455fd917ffcff7fdab80
I20260620 20:16:07.334458 20266 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 92c04115f7a84793bb5b64eecd929e90: Bootstrap starting.
I20260620 20:16:07.337814 20266 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 92c04115f7a84793bb5b64eecd929e90: Neither blocks nor log segments found. Creating new log.
I20260620 20:16:07.338768 20266 log.cc:824] T 00000000000000000000000000000000 P 92c04115f7a84793bb5b64eecd929e90: Log is configured to *not* fsync() on all Append() calls
Jun 20 20:16:07 dist-test-slave-2hpg krb5kdc[20186](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986566, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.217.62@KRBTEST.COM
I20260620 20:16:07.341614 20266 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 92c04115f7a84793bb5b64eecd929e90: No bootstrap required, opened a new log
I20260620 20:16:07.349382 20269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:07.326854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60368 (local address 127.16.217.62:32975)
0620 20:16:07.327657 (+ 803us) server_negotiation.cc:206] Beginning negotiation
0620 20:16:07.327675 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 20:16:07.327725 (+ 50us) server_negotiation.cc:407] Connection header received
0620 20:16:07.330005 (+ 2280us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:16:07.330033 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:16:07.330532 (+ 499us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:16:07.330973 (+ 441us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:16:07.332289 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:07.334178 (+ 1889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:07.335733 (+ 1555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:07.336770 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:07.336951 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:07.341086 (+ 4135us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:16:07.341096 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:16:07.341112 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:16:07.341160 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:16:07.345049 (+ 3889us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:16:07.345675 (+ 626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:16:07.345683 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:16:07.345691 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:16:07.345914 (+ 223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:16:07.346245 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:16:07.346250 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:16:07.346253 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:16:07.346897 (+ 644us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:16:07.347186 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0620 20:16:07.347573 (+ 387us) server_negotiation.cc:299] Negotiation successful
0620 20:16:07.347981 (+ 408us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":547,"thread_start_us":342,"threads_started":1}
I20260620 20:16:07.352185 20266 raft_consensus.cc:348] T 00000000000000000000000000000000 P 92c04115f7a84793bb5b64eecd929e90 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92c04115f7a84793bb5b64eecd929e90" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 32975 } }
I20260620 20:16:07.352612 20266 raft_consensus.cc:374] T 00000000000000000000000000000000 P 92c04115f7a84793bb5b64eecd929e90 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:16:07.352737 20266 raft_consensus.cc:729] T 00000000000000000000000000000000 P 92c04115f7a84793bb5b64eecd929e90 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92c04115f7a84793bb5b64eecd929e90, State: Initialized, Role: FOLLOWER
I20260620 20:16:07.353139 17252 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 20:16:07.353345 20266 consensus_queue.cc:260] T 00000000000000000000000000000000 P 92c04115f7a84793bb5b64eecd929e90 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92c04115f7a84793bb5b64eecd929e90" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 32975 } }
I20260620 20:16:07.353567 20266 raft_consensus.cc:388] T 00000000000000000000000000000000 P 92c04115f7a84793bb5b64eecd929e90 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 20:16:07.353714 20266 raft_consensus.cc:482] T 00000000000000000000000000000000 P 92c04115f7a84793bb5b64eecd929e90 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 20:16:07.353894 20266 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 92c04115f7a84793bb5b64eecd929e90 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:16:07.355885 20266 raft_consensus.cc:504] T 00000000000000000000000000000000 P 92c04115f7a84793bb5b64eecd929e90 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92c04115f7a84793bb5b64eecd929e90" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 32975 } }
I20260620 20:16:07.356345 20266 leader_election.cc:302] T 00000000000000000000000000000000 P 92c04115f7a84793bb5b64eecd929e90 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 92c04115f7a84793bb5b64eecd929e90; no voters:
I20260620 20:16:07.356904 20266 leader_election.cc:288] T 00000000000000000000000000000000 P 92c04115f7a84793bb5b64eecd929e90 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 20:16:07.357048 20271 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 92c04115f7a84793bb5b64eecd929e90 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:16:07.358842 20266 sys_catalog.cc:564] T 00000000000000000000000000000000 P 92c04115f7a84793bb5b64eecd929e90 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:16:07.360532 20271 raft_consensus.cc:686] T 00000000000000000000000000000000 P 92c04115f7a84793bb5b64eecd929e90 [term 1 LEADER]: Becoming Leader. State: Replica: 92c04115f7a84793bb5b64eecd929e90, State: Running, Role: LEADER
I20260620 20:16:07.361052 20271 consensus_queue.cc:237] T 00000000000000000000000000000000 P 92c04115f7a84793bb5b64eecd929e90 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92c04115f7a84793bb5b64eecd929e90" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 32975 } }
I20260620 20:16:07.366015 20273 sys_catalog.cc:455] T 00000000000000000000000000000000 P 92c04115f7a84793bb5b64eecd929e90 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 92c04115f7a84793bb5b64eecd929e90. Latest consensus state: current_term: 1 leader_uuid: "92c04115f7a84793bb5b64eecd929e90" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92c04115f7a84793bb5b64eecd929e90" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 32975 } } }
I20260620 20:16:07.366437 20273 sys_catalog.cc:458] T 00000000000000000000000000000000 P 92c04115f7a84793bb5b64eecd929e90 [sys.catalog]: This master's current role is: LEADER
I20260620 20:16:07.374851 20272 sys_catalog.cc:455] T 00000000000000000000000000000000 P 92c04115f7a84793bb5b64eecd929e90 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "92c04115f7a84793bb5b64eecd929e90" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92c04115f7a84793bb5b64eecd929e90" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 32975 } } }
I20260620 20:16:07.375172 20272 sys_catalog.cc:458] T 00000000000000000000000000000000 P 92c04115f7a84793bb5b64eecd929e90 [sys.catalog]: This master's current role is: LEADER
I20260620 20:16:07.375631 20283 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 20:16:07.380529 20283 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 20:16:07.390456 20283 catalog_manager.cc:1347] Generated new cluster ID: 26a0eea1b2114598aef626c003113ca7
I20260620 20:16:07.390589 20283 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 20:16:07.403558 20283 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 20:16:07.405016 20283 catalog_manager.cc:1504] Loading token signing keys...
I20260620 20:16:07.408254 20269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:07.399328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60376 (local address 127.16.217.62:32975)
0620 20:16:07.399492 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0620 20:16:07.399498 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:16:07.400061 (+ 563us) server_negotiation.cc:407] Connection header received
0620 20:16:07.400231 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:16:07.400237 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:16:07.400334 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:16:07.400466 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:16:07.401712 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:07.402680 (+ 968us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:07.404436 (+ 1756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:07.405147 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:07.405270 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:07.406505 (+ 1235us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:16:07.406512 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:16:07.406517 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:16:07.407409 (+ 892us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:16:07.407469 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:16:07.407495 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:16:07.407606 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0620 20:16:07.407752 (+ 146us) server_negotiation.cc:299] Negotiation successful
0620 20:16:07.407883 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":51}
I20260620 20:16:07.423134 20283 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 92c04115f7a84793bb5b64eecd929e90: Generated new TSK 0
I20260620 20:16:07.424186 20283 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 20:16:07.435189 17252 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu pid 20199
I20260620 20:16:07.436364 20261 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:16:07.524137 17252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu with pid 20199
2026-06-20T20:16:07Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1910 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986536687152-17252-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 20:16:07 dist-test-slave-2hpg krb5kdc[20301](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 20:16:07 dist-test-slave-2hpg krb5kdc[20301](info): set up 2 sockets
Jun 20 20:16:07 dist-test-slave-2hpg krb5kdc[20301](info): commencing operation
krb5kdc: starting...
W20260620 20:16:08.606526 17252 mini_kdc.cc:121] Time spent starting KDC: real 1.047s user 0.002s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986536687152-17252-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986536687152-17252-0/krb5kdc/test-user.keytab.
Jun 20 20:16:08 dist-test-slave-2hpg krb5kdc[20301](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986568, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T20:16:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T20:16:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.217.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.217.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.217.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.62.keytab.
Entry for principal kudu/127.16.217.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.62.keytab.
I20260620 20:16:08.794400 17252 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
/tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986536687152-17252-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986536687152-17252-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986536687152-17252-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986536687152-17252-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.217.62:37047
--webserver_interface=127.16.217.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.62.keytab
--principal=kudu/127.16.217.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.217.20:45793
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.217.62:37047
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986536687152-17252-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986536687152-17252-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986536687152-17252-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:16:09.106760 20314 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:16:09.107057 20314 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:16:09.107146 20314 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:16:09.114267 20314 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:16:09.114363 20314 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:16:09.114409 20314 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:16:09.114455 20314 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:16:09.114511 20314 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:16:09.124045 20314 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.217.20:45793
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986536687152-17252-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986536687152-17252-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.217.62:37047
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.62.keytab
--principal=kudu/127.16.217.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.217.62:37047
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986536687152-17252-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.217.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986536687152-17252-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:16:09.125272 20314 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:16:09.126500 20314 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:16:09.138576 20319 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:16:09.138648 20320 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:16:09.138604 20322 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:16:09.139604 20314 server_base.cc:1034] running on GCE node
I20260620 20:16:09.140466 20314 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:16:09.141974 20314 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:16:09.143208 20314 hybrid_clock.cc:648] HybridClock initialized: now 1781986569143153 us; error 48 us; skew 500 ppm
I20260620 20:16:09.143586 20314 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:16:09 dist-test-slave-2hpg krb5kdc[20301](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986569, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.217.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:16:09.146679 20314 init.cc:377] Logged in from keytab as kudu/127.16.217.62@KRBTEST.COM (short username kudu)
I20260620 20:16:09.148156 20314 webserver.cc:460] Webserver started at http://127.16.217.62:35479/ using document root <none> and password file <none>
I20260620 20:16:09.148825 20314 fs_manager.cc:359] Metadata directory not provided
I20260620 20:16:09.148969 20314 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:16:09.149266 20314 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:16:09.152204 20314 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986536687152-17252-0/minicluster-data/master-0/data/instance:
uuid: "5fa6b0fefb024d889590de82ed25af8a"
format_stamp: "Formatted at 2026-06-20 20:16:09 on dist-test-slave-2hpg"
server_key: "e07ed5cab713f3382c8108fe39fc329f"
server_key_iv: "c5de049f54632671cb919b52f64c0d43"
server_key_version: "encryptionkey@0"
I20260620 20:16:09.152943 20314 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986536687152-17252-0/minicluster-data/master-0/wal/instance:
uuid: "5fa6b0fefb024d889590de82ed25af8a"
format_stamp: "Formatted at 2026-06-20 20:16:09 on dist-test-slave-2hpg"
server_key: "e07ed5cab713f3382c8108fe39fc329f"
server_key_iv: "c5de049f54632671cb919b52f64c0d43"
server_key_version: "encryptionkey@0"
I20260620 20:16:09.158301 20314 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260620 20:16:09.161760 20329 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:16:09.163091 20314 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 20:16:09.163272 20314 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986536687152-17252-0/minicluster-data/master-0/data,/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986536687152-17252-0/minicluster-data/master-0/wal
uuid: "5fa6b0fefb024d889590de82ed25af8a"
format_stamp: "Formatted at 2026-06-20 20:16:09 on dist-test-slave-2hpg"
server_key: "e07ed5cab713f3382c8108fe39fc329f"
server_key_iv: "c5de049f54632671cb919b52f64c0d43"
server_key_version: "encryptionkey@0"
I20260620 20:16:09.163460 20314 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986536687152-17252-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986536687152-17252-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986536687152-17252-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:16:09.185823 20314 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:16:09.190371 20314 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:16:09.190699 20314 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:16:09.211267 20314 rpc_server.cc:307] RPC server started. Bound to: 127.16.217.62:37047
I20260620 20:16:09.211299 20380 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.217.62:37047 every 8 connection(s)
I20260620 20:16:09.212606 20314 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986536687152-17252-0/minicluster-data/master-0/data/info.pb
I20260620 20:16:09.216157 20381 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:16:09.222146 17252 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu as pid 20314
I20260620 20:16:09.222373 17252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986536687152-17252-0/minicluster-data/master-0/wal/instance
I20260620 20:16:09.223078 17252 external_mini_cluster.cc:1428] Setting key ca54ffe09d39d91206ab22d413d618b5
I20260620 20:16:09.226702 20381 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5fa6b0fefb024d889590de82ed25af8a: Bootstrap starting.
I20260620 20:16:09.230034 20381 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5fa6b0fefb024d889590de82ed25af8a: Neither blocks nor log segments found. Creating new log.
I20260620 20:16:09.231246 20381 log.cc:824] T 00000000000000000000000000000000 P 5fa6b0fefb024d889590de82ed25af8a: Log is configured to *not* fsync() on all Append() calls
I20260620 20:16:09.233953 20381 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5fa6b0fefb024d889590de82ed25af8a: No bootstrap required, opened a new log
Jun 20 20:16:09 dist-test-slave-2hpg krb5kdc[20301](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986568, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.217.62@KRBTEST.COM
I20260620 20:16:09.242506 20381 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5fa6b0fefb024d889590de82ed25af8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fa6b0fefb024d889590de82ed25af8a" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 37047 } }
I20260620 20:16:09.243024 20381 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5fa6b0fefb024d889590de82ed25af8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:16:09.243145 20381 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5fa6b0fefb024d889590de82ed25af8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fa6b0fefb024d889590de82ed25af8a, State: Initialized, Role: FOLLOWER
I20260620 20:16:09.243768 20381 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5fa6b0fefb024d889590de82ed25af8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fa6b0fefb024d889590de82ed25af8a" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 37047 } }
I20260620 20:16:09.244026 20381 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5fa6b0fefb024d889590de82ed25af8a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 20:16:09.244176 20381 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5fa6b0fefb024d889590de82ed25af8a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 20:16:09.244400 20381 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5fa6b0fefb024d889590de82ed25af8a [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:16:09.246316 20384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:09.225643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43454 (local address 127.16.217.62:37047)
0620 20:16:09.226541 (+ 898us) server_negotiation.cc:206] Beginning negotiation
0620 20:16:09.226554 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 20:16:09.226597 (+ 43us) server_negotiation.cc:407] Connection header received
0620 20:16:09.228381 (+ 1784us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:16:09.228409 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:16:09.228799 (+ 390us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:16:09.229119 (+ 320us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:16:09.230921 (+ 1802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:09.232328 (+ 1407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:09.233617 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:09.234419 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:09.234615 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:09.238896 (+ 4281us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:16:09.238904 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:16:09.238918 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:16:09.238955 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:16:09.242029 (+ 3074us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:16:09.242811 (+ 782us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:16:09.242817 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:16:09.242824 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:16:09.243042 (+ 218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:16:09.243472 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:16:09.243479 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:16:09.243485 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:16:09.244135 (+ 650us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:16:09.244454 (+ 319us) server_negotiation.cc:1035] Waiting for connection context
0620 20:16:09.244822 (+ 368us) server_negotiation.cc:299] Negotiation successful
0620 20:16:09.245172 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":684,"thread_start_us":461,"threads_started":1}
I20260620 20:16:09.247054 20381 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5fa6b0fefb024d889590de82ed25af8a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fa6b0fefb024d889590de82ed25af8a" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 37047 } }
I20260620 20:16:09.247538 20381 leader_election.cc:302] T 00000000000000000000000000000000 P 5fa6b0fefb024d889590de82ed25af8a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5fa6b0fefb024d889590de82ed25af8a; no voters:
I20260620 20:16:09.248131 20381 leader_election.cc:288] T 00000000000000000000000000000000 P 5fa6b0fefb024d889590de82ed25af8a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 20:16:09.248274 20386 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5fa6b0fefb024d889590de82ed25af8a [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:16:09.249349 20386 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5fa6b0fefb024d889590de82ed25af8a [term 1 LEADER]: Becoming Leader. State: Replica: 5fa6b0fefb024d889590de82ed25af8a, State: Running, Role: LEADER
I20260620 20:16:09.249979 20386 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5fa6b0fefb024d889590de82ed25af8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fa6b0fefb024d889590de82ed25af8a" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 37047 } }
I20260620 20:16:09.250365 20381 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5fa6b0fefb024d889590de82ed25af8a [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:16:09.252477 20388 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5fa6b0fefb024d889590de82ed25af8a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5fa6b0fefb024d889590de82ed25af8a. Latest consensus state: current_term: 1 leader_uuid: "5fa6b0fefb024d889590de82ed25af8a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fa6b0fefb024d889590de82ed25af8a" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 37047 } } }
I20260620 20:16:09.252812 20388 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5fa6b0fefb024d889590de82ed25af8a [sys.catalog]: This master's current role is: LEADER
I20260620 20:16:09.253365 20387 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5fa6b0fefb024d889590de82ed25af8a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5fa6b0fefb024d889590de82ed25af8a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fa6b0fefb024d889590de82ed25af8a" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 37047 } } }
I20260620 20:16:09.253737 20387 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5fa6b0fefb024d889590de82ed25af8a [sys.catalog]: This master's current role is: LEADER
I20260620 20:16:09.260499 20393 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 20:16:09.267588 20393 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 20:16:09.277029 20393 catalog_manager.cc:1347] Generated new cluster ID: d85e35c84ed34120a8b1b4c2e1f19783
I20260620 20:16:09.277143 20393 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 20:16:09.286298 20393 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 20:16:09.288146 20393 catalog_manager.cc:1504] Loading token signing keys...
I20260620 20:16:09.298915 20393 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5fa6b0fefb024d889590de82ed25af8a: Generated new TSK 0
I20260620 20:16:09.300052 20393 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.217.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.217.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.217.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.1.keytab.
Entry for principal kudu/127.16.217.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.1.keytab.
I20260620 20:16:09.349821 17252 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
/tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986536687152-17252-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986536687152-17252-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986536687152-17252-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986536687152-17252-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.217.1:0
--local_ip_for_outbound_sockets=127.16.217.1
--webserver_interface=127.16.217.1
--webserver_port=0
--tserver_master_addrs=127.16.217.62:37047
--keytab_file=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.1.keytab
--principal=kudu/127.16.217.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.217.20:45793
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986536687152-17252-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986536687152-17252-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986536687152-17252-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:16:09.670336 20408 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:16:09.670589 20408 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:16:09.670737 20408 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:16:09.677788 20408 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:16:09.677953 20408 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:16:09.678149 20408 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.217.1
I20260620 20:16:09.687475 20408 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.217.20:45793
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986536687152-17252-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986536687152-17252-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.1.keytab
--principal=kudu/127.16.217.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.217.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986536687152-17252-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.217.1
--webserver_port=0
--tserver_master_addrs=127.16.217.62:37047
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.217.1
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986536687152-17252-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:16:09.688675 20408 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:16:09.690065 20408 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:16:09.703395 20413 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:16:09.703459 20414 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:16:09.703492 20416 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:16:09.704514 20408 server_base.cc:1034] running on GCE node
I20260620 20:16:09.705152 20408 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:16:09.706104 20408 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:16:09.707358 20408 hybrid_clock.cc:648] HybridClock initialized: now 1781986569707307 us; error 57 us; skew 500 ppm
I20260620 20:16:09.707793 20408 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:16:09 dist-test-slave-2hpg krb5kdc[20301](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986569, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.217.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:16:09.710870 20408 init.cc:377] Logged in from keytab as kudu/127.16.217.1@KRBTEST.COM (short username kudu)
I20260620 20:16:09.712271 20408 webserver.cc:460] Webserver started at http://127.16.217.1:43451/ using document root <none> and password file <none>
I20260620 20:16:09.712947 20408 fs_manager.cc:359] Metadata directory not provided
I20260620 20:16:09.713093 20408 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:16:09.713392 20408 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:16:09.716225 20408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986536687152-17252-0/minicluster-data/ts-0/data/instance:
uuid: "b59313ae28b7432097d1608b64e87db3"
format_stamp: "Formatted at 2026-06-20 20:16:09 on dist-test-slave-2hpg"
server_key: "5f2f3c92814bffbca49f6ca5c01bd979"
server_key_iv: "c51c406f7f622fd148e7f229e85be1df"
server_key_version: "encryptionkey@0"
I20260620 20:16:09.716985 20408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986536687152-17252-0/minicluster-data/ts-0/wal/instance:
uuid: "b59313ae28b7432097d1608b64e87db3"
format_stamp: "Formatted at 2026-06-20 20:16:09 on dist-test-slave-2hpg"
server_key: "5f2f3c92814bffbca49f6ca5c01bd979"
server_key_iv: "c51c406f7f622fd148e7f229e85be1df"
server_key_version: "encryptionkey@0"
I20260620 20:16:09.721961 20408 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260620 20:16:09.725180 20423 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:16:09.726475 20408 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260620 20:16:09.726637 20408 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986536687152-17252-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986536687152-17252-0/minicluster-data/ts-0/wal
uuid: "b59313ae28b7432097d1608b64e87db3"
format_stamp: "Formatted at 2026-06-20 20:16:09 on dist-test-slave-2hpg"
server_key: "5f2f3c92814bffbca49f6ca5c01bd979"
server_key_iv: "c51c406f7f622fd148e7f229e85be1df"
server_key_version: "encryptionkey@0"
I20260620 20:16:09.726843 20408 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986536687152-17252-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986536687152-17252-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986536687152-17252-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:16:09.744671 20408 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:16:09.748993 20408 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:16:09.749262 20408 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:16:09.750272 20408 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:16:09.751562 20408 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:16:09.751683 20408 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:16:09.751835 20408 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:16:09.751904 20408 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:16:09.785356 20408 rpc_server.cc:307] RPC server started. Bound to: 127.16.217.1:36549
I20260620 20:16:09.785413 20535 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.217.1:36549 every 8 connection(s)
I20260620 20:16:09.786722 20408 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986536687152-17252-0/minicluster-data/ts-0/data/info.pb
I20260620 20:16:09.789239 17252 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu as pid 20408
I20260620 20:16:09.789462 17252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986536687152-17252-0/minicluster-data/ts-0/wal/instance
I20260620 20:16:09.790275 17252 external_mini_cluster.cc:1428] Setting key 750516b8ab61d5968eb5468fea31f353
Jun 20 20:16:09 dist-test-slave-2hpg krb5kdc[20301](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986569, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.217.1@KRBTEST.COM for kudu/127.16.217.62@KRBTEST.COM
I20260620 20:16:09.807161 20538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:09.789291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.217.1:48601 (local address 127.16.217.62:37047)
0620 20:16:09.789784 (+ 493us) server_negotiation.cc:206] Beginning negotiation
0620 20:16:09.789791 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:16:09.790887 (+ 1096us) server_negotiation.cc:407] Connection header received
0620 20:16:09.792957 (+ 2070us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:16:09.792965 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:16:09.793051 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:16:09.793171 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:16:09.794674 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:09.795850 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:09.797467 (+ 1617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:09.797925 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:09.798091 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:09.801480 (+ 3389us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:16:09.801486 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:16:09.801498 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:16:09.801528 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:16:09.804203 (+ 2675us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:16:09.804915 (+ 712us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:16:09.804923 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:16:09.804928 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:16:09.805052 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:16:09.805462 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:16:09.805469 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:16:09.805472 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:16:09.805739 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:16:09.805902 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0620 20:16:09.806572 (+ 670us) server_negotiation.cc:299] Negotiation successful
0620 20:16:09.806780 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":343,"thread_start_us":220,"threads_started":1}
WARNING: no policy specified for kudu/127.16.217.2@KRBTEST.COM; defaulting to no policy
I20260620 20:16:09.808187 20539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:09.789849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:37047 (local address 127.16.217.1:48601)
0620 20:16:09.790703 (+ 854us) negotiation.cc:107] Waiting for socket to connect
0620 20:16:09.790755 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0620 20:16:09.792685 (+ 1930us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:16:09.793382 (+ 697us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:16:09.793390 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:16:09.793937 (+ 547us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:16:09.794508 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:16:09.794523 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:16:09.796025 (+ 1502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:16:09.796029 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:16:09.797324 (+ 1295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:16:09.797335 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:16:09.798164 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:16:09.798171 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:16:09.798402 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:09.799175 (+ 773us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:16:09.799201 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:16:09.801260 (+ 2059us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:16:09.804360 (+ 3100us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:16:09.804369 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:16:09.804386 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:16:09.804796 (+ 410us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:16:09.805160 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:16:09.805166 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:16:09.805169 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:16:09.805355 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:16:09.805885 (+ 530us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:16:09.805894 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:16:09.806244 (+ 350us) client_negotiation.cc:769] Sending connection context
0620 20:16:09.806497 (+ 253us) client_negotiation.cc:240] Negotiation successful
0620 20:16:09.807043 (+ 546us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":535,"thread_start_us":335,"threads_started":1}
I20260620 20:16:09.809695 20536 heartbeater.cc:347] Connected to a master server at 127.16.217.62:37047
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.217.2@KRBTEST.COM" created.
I20260620 20:16:09.810050 20536 heartbeater.cc:464] Registering TS with master...
I20260620 20:16:09.810937 20536 heartbeater.cc:511] Master 127.16.217.62:37047 requested a full tablet report, sending...
I20260620 20:16:09.813095 20346 ts_manager.cc:194] Registered new tserver with Master: b59313ae28b7432097d1608b64e87db3 (127.16.217.1:36549)
I20260620 20:16:09.814567 20346 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.217.1@KRBTEST.COM'} at 127.16.217.1:48601
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.217.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.2.keytab.
Entry for principal kudu/127.16.217.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.2.keytab.
I20260620 20:16:09.832675 17252 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
/tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986536687152-17252-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986536687152-17252-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986536687152-17252-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986536687152-17252-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.217.2:0
--local_ip_for_outbound_sockets=127.16.217.2
--webserver_interface=127.16.217.2
--webserver_port=0
--tserver_master_addrs=127.16.217.62:37047
--keytab_file=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.2.keytab
--principal=kudu/127.16.217.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.217.20:45793
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986536687152-17252-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986536687152-17252-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986536687152-17252-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:16:10.148495 20542 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:16:10.148780 20542 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:16:10.148883 20542 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:16:10.156292 20542 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:16:10.156404 20542 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:16:10.156517 20542 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.217.2
I20260620 20:16:10.166465 20542 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.217.20:45793
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986536687152-17252-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986536687152-17252-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.2.keytab
--principal=kudu/127.16.217.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.217.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986536687152-17252-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.217.2
--webserver_port=0
--tserver_master_addrs=127.16.217.62:37047
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.217.2
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986536687152-17252-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:16:10.167773 20542 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:16:10.169281 20542 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:16:10.182183 20547 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:16:10.182137 20548 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:16:10.182219 20550 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:16:10.183007 20542 server_base.cc:1034] running on GCE node
I20260620 20:16:10.183647 20542 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:16:10.184580 20542 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:16:10.185793 20542 hybrid_clock.cc:648] HybridClock initialized: now 1781986570185747 us; error 38 us; skew 500 ppm
I20260620 20:16:10.186167 20542 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:16:10 dist-test-slave-2hpg krb5kdc[20301](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986570, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.217.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:16:10.189453 20542 init.cc:377] Logged in from keytab as kudu/127.16.217.2@KRBTEST.COM (short username kudu)
I20260620 20:16:10.190958 20542 webserver.cc:460] Webserver started at http://127.16.217.2:37865/ using document root <none> and password file <none>
I20260620 20:16:10.191689 20542 fs_manager.cc:359] Metadata directory not provided
I20260620 20:16:10.191847 20542 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:16:10.192162 20542 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:16:10.195195 20542 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986536687152-17252-0/minicluster-data/ts-1/data/instance:
uuid: "1d092f32e76c400d850b6792acc1fd00"
format_stamp: "Formatted at 2026-06-20 20:16:10 on dist-test-slave-2hpg"
server_key: "d6a8e00ef507134f93b8456f652a525a"
server_key_iv: "15b5e6fb3a3bf7da2701f05f6f1005d0"
server_key_version: "encryptionkey@0"
I20260620 20:16:10.196019 20542 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986536687152-17252-0/minicluster-data/ts-1/wal/instance:
uuid: "1d092f32e76c400d850b6792acc1fd00"
format_stamp: "Formatted at 2026-06-20 20:16:10 on dist-test-slave-2hpg"
server_key: "d6a8e00ef507134f93b8456f652a525a"
server_key_iv: "15b5e6fb3a3bf7da2701f05f6f1005d0"
server_key_version: "encryptionkey@0"
I20260620 20:16:10.201745 20542 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260620 20:16:10.205165 20557 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:16:10.206527 20542 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260620 20:16:10.206696 20542 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986536687152-17252-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986536687152-17252-0/minicluster-data/ts-1/wal
uuid: "1d092f32e76c400d850b6792acc1fd00"
format_stamp: "Formatted at 2026-06-20 20:16:10 on dist-test-slave-2hpg"
server_key: "d6a8e00ef507134f93b8456f652a525a"
server_key_iv: "15b5e6fb3a3bf7da2701f05f6f1005d0"
server_key_version: "encryptionkey@0"
I20260620 20:16:10.206913 20542 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986536687152-17252-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986536687152-17252-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986536687152-17252-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:16:10.218977 20542 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:16:10.223085 20542 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:16:10.223376 20542 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:16:10.224241 20542 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:16:10.225584 20542 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:16:10.225684 20542 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:16:10.225833 20542 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:16:10.225925 20542 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260620 20:16:10.258627 20542 rpc_server.cc:307] RPC server started. Bound to: 127.16.217.2:40251
I20260620 20:16:10.258675 20670 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.217.2:40251 every 8 connection(s)
I20260620 20:16:10.259931 20542 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986536687152-17252-0/minicluster-data/ts-1/data/info.pb
I20260620 20:16:10.260306 17252 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu as pid 20542
I20260620 20:16:10.260504 17252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986536687152-17252-0/minicluster-data/ts-1/wal/instance
I20260620 20:16:10.261122 17252 external_mini_cluster.cc:1428] Setting key fc82ca24df2d3965b9926f454f007870
Jun 20 20:16:10 dist-test-slave-2hpg krb5kdc[20301](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986570, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.217.2@KRBTEST.COM for kudu/127.16.217.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.16.217.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.217.3@KRBTEST.COM" created.
I20260620 20:16:10.281952 20538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:10.262928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.217.2:38527 (local address 127.16.217.62:37047)
0620 20:16:10.263089 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0620 20:16:10.263094 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:16:10.264585 (+ 1491us) server_negotiation.cc:407] Connection header received
0620 20:16:10.266454 (+ 1869us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:16:10.266462 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:16:10.266558 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:16:10.266676 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:16:10.268222 (+ 1546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:10.269113 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:10.271176 (+ 2063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:10.271623 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:10.271792 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:10.275637 (+ 3845us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:16:10.275643 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:16:10.275647 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:16:10.275679 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:16:10.278624 (+ 2945us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:16:10.279482 (+ 858us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:16:10.279495 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:16:10.279498 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:16:10.279623 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:16:10.280124 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:16:10.280127 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:16:10.280130 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:16:10.280422 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:16:10.280540 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0620 20:16:10.281401 (+ 861us) server_negotiation.cc:299] Negotiation successful
0620 20:16:10.281577 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 20:16:10.283035 20673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:10.263593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:37047 (local address 127.16.217.2:38527)
0620 20:16:10.264406 (+ 813us) negotiation.cc:107] Waiting for socket to connect
0620 20:16:10.264457 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 20:16:10.266180 (+ 1723us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:16:10.266903 (+ 723us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:16:10.266911 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:16:10.267470 (+ 559us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:16:10.268034 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:16:10.268051 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:16:10.269328 (+ 1277us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:16:10.269334 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:16:10.271020 (+ 1686us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:16:10.271033 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:16:10.271855 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:16:10.271864 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:16:10.272104 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:10.272878 (+ 774us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:16:10.272900 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:16:10.275387 (+ 2487us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:16:10.278781 (+ 3394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:16:10.278792 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:16:10.278810 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:16:10.279349 (+ 539us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:16:10.279740 (+ 391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:16:10.279746 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:16:10.279751 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:16:10.279988 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:16:10.280565 (+ 577us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:16:10.280571 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:16:10.280919 (+ 348us) client_negotiation.cc:769] Sending connection context
0620 20:16:10.281287 (+ 368us) client_negotiation.cc:240] Negotiation successful
0620 20:16:10.281782 (+ 495us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":545,"thread_start_us":341,"threads_started":1}
I20260620 20:16:10.284794 20671 heartbeater.cc:347] Connected to a master server at 127.16.217.62:37047
I20260620 20:16:10.285145 20671 heartbeater.cc:464] Registering TS with master...
I20260620 20:16:10.285856 20671 heartbeater.cc:511] Master 127.16.217.62:37047 requested a full tablet report, sending...
I20260620 20:16:10.287416 20346 ts_manager.cc:194] Registered new tserver with Master: 1d092f32e76c400d850b6792acc1fd00 (127.16.217.2:40251)
I20260620 20:16:10.288246 20346 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.217.2@KRBTEST.COM'} at 127.16.217.2:38527
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.217.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.3.keytab.
Entry for principal kudu/127.16.217.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.3.keytab.
I20260620 20:16:10.306135 17252 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
/tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986536687152-17252-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986536687152-17252-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986536687152-17252-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986536687152-17252-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.217.3:0
--local_ip_for_outbound_sockets=127.16.217.3
--webserver_interface=127.16.217.3
--webserver_port=0
--tserver_master_addrs=127.16.217.62:37047
--keytab_file=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.3.keytab
--principal=kudu/127.16.217.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.217.20:45793
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986536687152-17252-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986536687152-17252-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986536687152-17252-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:16:10.630523 20676 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:16:10.630820 20676 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:16:10.630918 20676 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:16:10.638216 20676 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:16:10.638334 20676 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:16:10.638455 20676 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.217.3
I20260620 20:16:10.648092 20676 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.217.20:45793
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986536687152-17252-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986536687152-17252-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.3.keytab
--principal=kudu/127.16.217.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.217.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986536687152-17252-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.217.3
--webserver_port=0
--tserver_master_addrs=127.16.217.62:37047
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.217.3
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986536687152-17252-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:16:10.649346 20676 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:16:10.650789 20676 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:16:10.664444 20681 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:16:10.664496 20684 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:16:10.664538 20682 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:16:10.665473 20676 server_base.cc:1034] running on GCE node
I20260620 20:16:10.666114 20676 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:16:10.667069 20676 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:16:10.668296 20676 hybrid_clock.cc:648] HybridClock initialized: now 1781986570668247 us; error 30 us; skew 500 ppm
I20260620 20:16:10.668649 20676 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:16:10 dist-test-slave-2hpg krb5kdc[20301](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986570, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.217.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:16:10.671850 20676 init.cc:377] Logged in from keytab as kudu/127.16.217.3@KRBTEST.COM (short username kudu)
I20260620 20:16:10.673283 20676 webserver.cc:460] Webserver started at http://127.16.217.3:35177/ using document root <none> and password file <none>
I20260620 20:16:10.674019 20676 fs_manager.cc:359] Metadata directory not provided
I20260620 20:16:10.674180 20676 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:16:10.674470 20676 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:16:10.677259 20676 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986536687152-17252-0/minicluster-data/ts-2/data/instance:
uuid: "cb355edf6f5343ba9b7a958120ecdb7c"
format_stamp: "Formatted at 2026-06-20 20:16:10 on dist-test-slave-2hpg"
server_key: "4d06762e84ce9430db8086747c8031c0"
server_key_iv: "98f6b4d8a93f35d50402a463d1bcd163"
server_key_version: "encryptionkey@0"
I20260620 20:16:10.678062 20676 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986536687152-17252-0/minicluster-data/ts-2/wal/instance:
uuid: "cb355edf6f5343ba9b7a958120ecdb7c"
format_stamp: "Formatted at 2026-06-20 20:16:10 on dist-test-slave-2hpg"
server_key: "4d06762e84ce9430db8086747c8031c0"
server_key_iv: "98f6b4d8a93f35d50402a463d1bcd163"
server_key_version: "encryptionkey@0"
I20260620 20:16:10.682883 20676 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260620 20:16:10.686237 20691 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:16:10.687454 20676 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260620 20:16:10.687616 20676 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986536687152-17252-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986536687152-17252-0/minicluster-data/ts-2/wal
uuid: "cb355edf6f5343ba9b7a958120ecdb7c"
format_stamp: "Formatted at 2026-06-20 20:16:10 on dist-test-slave-2hpg"
server_key: "4d06762e84ce9430db8086747c8031c0"
server_key_iv: "98f6b4d8a93f35d50402a463d1bcd163"
server_key_version: "encryptionkey@0"
I20260620 20:16:10.687803 20676 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986536687152-17252-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986536687152-17252-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986536687152-17252-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:16:10.701941 20676 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:16:10.705819 20676 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:16:10.706118 20676 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:16:10.706996 20676 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:16:10.708287 20676 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:16:10.708395 20676 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:16:10.708529 20676 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:16:10.708597 20676 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:16:10.740080 20676 rpc_server.cc:307] RPC server started. Bound to: 127.16.217.3:39995
I20260620 20:16:10.740123 20803 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.217.3:39995 every 8 connection(s)
I20260620 20:16:10.741338 20676 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986536687152-17252-0/minicluster-data/ts-2/data/info.pb
I20260620 20:16:10.744441 17252 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu as pid 20676
I20260620 20:16:10.744617 17252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986536687152-17252-0/minicluster-data/ts-2/wal/instance
I20260620 20:16:10.745175 17252 external_mini_cluster.cc:1428] Setting key 672c5c04aee4be1af1aaac5e56aa1bea
Jun 20 20:16:10 dist-test-slave-2hpg krb5kdc[20301](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986570, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.217.3@KRBTEST.COM for kudu/127.16.217.62@KRBTEST.COM
I20260620 20:16:10.761067 20538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:10.743790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.217.3:48451 (local address 127.16.217.62:37047)
0620 20:16:10.743951 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0620 20:16:10.743956 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:16:10.745325 (+ 1369us) server_negotiation.cc:407] Connection header received
0620 20:16:10.747340 (+ 2015us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:16:10.747346 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:16:10.747433 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:16:10.747530 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:16:10.749056 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:10.749776 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:10.751328 (+ 1552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:10.751749 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:10.751880 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:10.755064 (+ 3184us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:16:10.755069 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:16:10.755072 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:16:10.755102 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:16:10.757774 (+ 2672us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:16:10.758571 (+ 797us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:16:10.758581 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:16:10.758584 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:16:10.758701 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:16:10.759244 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:16:10.759249 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:16:10.759252 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:16:10.759567 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:16:10.759723 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0620 20:16:10.760498 (+ 775us) server_negotiation.cc:299] Negotiation successful
0620 20:16:10.760702 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 20:16:10.762187 20806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:10.744326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:37047 (local address 127.16.217.3:48451)
0620 20:16:10.745120 (+ 794us) negotiation.cc:107] Waiting for socket to connect
0620 20:16:10.745158 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 20:16:10.747084 (+ 1926us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:16:10.747759 (+ 675us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:16:10.747769 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:16:10.748364 (+ 595us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:16:10.748907 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:16:10.748924 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:16:10.749951 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:16:10.749955 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:16:10.751188 (+ 1233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:16:10.751199 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:16:10.751960 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:16:10.751970 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:16:10.752212 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:10.752783 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:16:10.752804 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:16:10.754896 (+ 2092us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:16:10.757955 (+ 3059us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:16:10.757961 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:16:10.757973 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:16:10.758440 (+ 467us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:16:10.758853 (+ 413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:16:10.758856 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:16:10.758859 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:16:10.759119 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:16:10.759697 (+ 578us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:16:10.759705 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:16:10.760066 (+ 361us) client_negotiation.cc:769] Sending connection context
0620 20:16:10.760389 (+ 323us) client_negotiation.cc:240] Negotiation successful
0620 20:16:10.760948 (+ 559us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":548,"thread_start_us":345,"threads_started":1}
I20260620 20:16:10.763412 20804 heartbeater.cc:347] Connected to a master server at 127.16.217.62:37047
I20260620 20:16:10.763659 20804 heartbeater.cc:464] Registering TS with master...
I20260620 20:16:10.764312 20804 heartbeater.cc:511] Master 127.16.217.62:37047 requested a full tablet report, sending...
I20260620 20:16:10.765554 20346 ts_manager.cc:194] Registered new tserver with Master: cb355edf6f5343ba9b7a958120ecdb7c (127.16.217.3:39995)
I20260620 20:16:10.766314 20346 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.217.3@KRBTEST.COM'} at 127.16.217.3:48451
I20260620 20:16:10.772032 17252 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 20:16:10 dist-test-slave-2hpg krb5kdc[20301](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986570, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Jun 20 20:16:10 dist-test-slave-2hpg krb5kdc[20301](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986570, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.16.217.62@KRBTEST.COM
I20260620 20:16:10.817384 20536 heartbeater.cc:503] Master 127.16.217.62:37047 was elected leader, sending a full tablet report...
I20260620 20:16:10.820833 20538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:10.806889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42756 (local address 127.16.217.62:37047)
0620 20:16:10.807143 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0620 20:16:10.807150 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:16:10.807444 (+ 294us) server_negotiation.cc:407] Connection header received
0620 20:16:10.807595 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:16:10.807598 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:16:10.807662 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:16:10.807746 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:16:10.809188 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:10.810043 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:10.811316 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:10.811759 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:10.811856 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:10.815525 (+ 3669us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:16:10.815531 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:16:10.815534 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:16:10.815564 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:16:10.818351 (+ 2787us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:16:10.818960 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:16:10.818969 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:16:10.818973 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:16:10.819110 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:16:10.819472 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:16:10.819479 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:16:10.819482 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:16:10.819793 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:16:10.820028 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0620 20:16:10.820334 (+ 306us) server_negotiation.cc:299] Negotiation successful
0620 20:16:10.820541 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
W20260620 20:16:10.821322 20346 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:42756
I20260620 20:16:10.822845 17252 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu pid 20408
Jun 20 20:16:10 dist-test-slave-2hpg krb5kdc[20301](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986570, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.16.217.1@KRBTEST.COM
I20260620 20:16:10.837314 20815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:10.823909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46352 (local address 127.16.217.1:36549)
0620 20:16:10.824452 (+ 543us) server_negotiation.cc:206] Beginning negotiation
0620 20:16:10.824459 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:16:10.824483 (+ 24us) server_negotiation.cc:407] Connection header received
0620 20:16:10.824529 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:16:10.824534 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:16:10.824735 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:16:10.824857 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:16:10.826071 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:10.827038 (+ 967us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:10.828178 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:10.828743 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:10.828840 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:10.832563 (+ 3723us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:16:10.832574 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:16:10.832587 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:16:10.832617 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:16:10.835118 (+ 2501us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:16:10.835731 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:16:10.835738 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:16:10.835740 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:16:10.835861 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:16:10.836147 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:16:10.836150 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:16:10.836152 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:16:10.836471 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:16:10.836678 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0620 20:16:10.836853 (+ 175us) server_negotiation.cc:299] Negotiation successful
0620 20:16:10.837033 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":405,"thread_start_us":263,"threads_started":1}
W20260620 20:16:10.838570 20531 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:46352
W20260620 20:16:10.839540 17252 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu pid 20408: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 20:16:10.839735 17252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu with pid 20408
I20260620 20:16:10.854990 17252 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu pid 20542
Jun 20 20:16:10 dist-test-slave-2hpg krb5kdc[20301](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986570, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.16.217.2@KRBTEST.COM
I20260620 20:16:10.870371 20816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:10.855749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50886 (local address 127.16.217.2:40251)
0620 20:16:10.856231 (+ 482us) server_negotiation.cc:206] Beginning negotiation
0620 20:16:10.856239 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 20:16:10.856262 (+ 23us) server_negotiation.cc:407] Connection header received
0620 20:16:10.856317 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:16:10.856323 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:16:10.856523 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:16:10.856683 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:16:10.857848 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:10.858941 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:10.860119 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:10.860681 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:10.860822 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:10.864928 (+ 4106us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:16:10.864939 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:16:10.864953 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:16:10.864984 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:16:10.867650 (+ 2666us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:16:10.868316 (+ 666us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:16:10.868325 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:16:10.868327 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:16:10.868436 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:16:10.868844 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:16:10.868847 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:16:10.868849 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:16:10.869144 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:16:10.869373 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0620 20:16:10.869674 (+ 301us) server_negotiation.cc:299] Negotiation successful
0620 20:16:10.869942 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":356,"thread_start_us":233,"threads_started":1}
W20260620 20:16:10.871630 20666 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:50886
W20260620 20:16:10.872835 17252 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu pid 20542: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 20:16:10.872954 17252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu with pid 20542
I20260620 20:16:10.889293 17252 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu pid 20676
Jun 20 20:16:10 dist-test-slave-2hpg krb5kdc[20301](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986570, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.16.217.3@KRBTEST.COM
I20260620 20:16:10.905148 20817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:10.890250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34460 (local address 127.16.217.3:39995)
0620 20:16:10.890874 (+ 624us) server_negotiation.cc:206] Beginning negotiation
0620 20:16:10.890884 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 20:16:10.890920 (+ 36us) server_negotiation.cc:407] Connection header received
0620 20:16:10.890982 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:16:10.890988 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:16:10.891198 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:16:10.891391 (+ 193us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:16:10.892795 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:10.893836 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:10.895164 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:10.895736 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:10.895868 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:10.899919 (+ 4051us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:16:10.899929 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:16:10.899944 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:16:10.899977 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:16:10.902793 (+ 2816us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:16:10.903369 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:16:10.903376 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:16:10.903379 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:16:10.903492 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:16:10.903807 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:16:10.903811 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:16:10.903813 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:16:10.904139 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:16:10.904345 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0620 20:16:10.904567 (+ 222us) server_negotiation.cc:299] Negotiation successful
0620 20:16:10.904749 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":443,"thread_start_us":288,"threads_started":1}
W20260620 20:16:10.906477 20799 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:34460
W20260620 20:16:10.907606 17252 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu pid 20676: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 20:16:10.907794 17252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu with pid 20676
I20260620 20:16:10.922952 17252 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu pid 20314
I20260620 20:16:10.924096 20376 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:16:11.024734 17252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu with pid 20314
2026-06-20T20:16:11Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (3505 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986536687152-17252-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 20:16:11 dist-test-slave-2hpg krb5kdc[20821](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 20:16:11 dist-test-slave-2hpg krb5kdc[20821](info): set up 2 sockets
Jun 20 20:16:11 dist-test-slave-2hpg krb5kdc[20821](info): commencing operation
krb5kdc: starting...
W20260620 20:16:12.118628 17252 mini_kdc.cc:121] Time spent starting KDC: real 1.053s user 0.002s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986536687152-17252-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986536687152-17252-0/krb5kdc/test-user.keytab.
Jun 20 20:16:12 dist-test-slave-2hpg krb5kdc[20821](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986572, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T20:16:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T20:16:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.217.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.217.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.217.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.62.keytab.
Entry for principal kudu/127.16.217.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.62.keytab.
I20260620 20:16:12.307713 17252 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
/tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986536687152-17252-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986536687152-17252-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986536687152-17252-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986536687152-17252-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.217.62:35261
--webserver_interface=127.16.217.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.62.keytab
--principal=kudu/127.16.217.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.217.20:43717
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.217.62:35261
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986536687152-17252-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986536687152-17252-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986536687152-17252-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:16:12.616827 20834 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:16:12.617146 20834 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:16:12.617312 20834 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:16:12.625212 20834 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:16:12.625360 20834 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:16:12.625483 20834 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:16:12.625600 20834 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:16:12.625691 20834 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:16:12.635780 20834 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.217.20:43717
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986536687152-17252-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986536687152-17252-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.217.62:35261
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.62.keytab
--principal=kudu/127.16.217.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.217.62:35261
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986536687152-17252-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.217.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986536687152-17252-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:16:12.637120 20834 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:16:12.638500 20834 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:16:12.650437 20842 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:16:12.650424 20840 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:16:12.650472 20839 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:16:12.651554 20834 server_base.cc:1034] running on GCE node
I20260620 20:16:12.652315 20834 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:16:12.653744 20834 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:16:12.654991 20834 hybrid_clock.cc:648] HybridClock initialized: now 1781986572654937 us; error 63 us; skew 500 ppm
I20260620 20:16:12.655350 20834 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:16:12 dist-test-slave-2hpg krb5kdc[20821](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986572, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.217.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:16:12.663254 20834 init.cc:377] Logged in from keytab as kudu/127.16.217.62@KRBTEST.COM (short username kudu)
I20260620 20:16:12.664623 20834 webserver.cc:460] Webserver started at http://127.16.217.62:44837/ using document root <none> and password file <none>
I20260620 20:16:12.665287 20834 fs_manager.cc:359] Metadata directory not provided
I20260620 20:16:12.665385 20834 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:16:12.665694 20834 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:16:12.668625 20834 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986536687152-17252-0/minicluster-data/master-0/data/instance:
uuid: "60904aed11334c78ab2c878b5701390a"
format_stamp: "Formatted at 2026-06-20 20:16:12 on dist-test-slave-2hpg"
server_key: "3dcdeb503ca7dfd50f3bfb590d01c49b"
server_key_iv: "8c63f3f2e56100de9192d1b3b01a6004"
server_key_version: "encryptionkey@0"
I20260620 20:16:12.669399 20834 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986536687152-17252-0/minicluster-data/master-0/wal/instance:
uuid: "60904aed11334c78ab2c878b5701390a"
format_stamp: "Formatted at 2026-06-20 20:16:12 on dist-test-slave-2hpg"
server_key: "3dcdeb503ca7dfd50f3bfb590d01c49b"
server_key_iv: "8c63f3f2e56100de9192d1b3b01a6004"
server_key_version: "encryptionkey@0"
I20260620 20:16:12.674665 20834 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260620 20:16:12.678534 20849 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:16:12.679834 20834 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 20:16:12.680008 20834 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986536687152-17252-0/minicluster-data/master-0/data,/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986536687152-17252-0/minicluster-data/master-0/wal
uuid: "60904aed11334c78ab2c878b5701390a"
format_stamp: "Formatted at 2026-06-20 20:16:12 on dist-test-slave-2hpg"
server_key: "3dcdeb503ca7dfd50f3bfb590d01c49b"
server_key_iv: "8c63f3f2e56100de9192d1b3b01a6004"
server_key_version: "encryptionkey@0"
I20260620 20:16:12.680187 20834 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986536687152-17252-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986536687152-17252-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986536687152-17252-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:16:12.719612 20834 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:16:12.724085 20834 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:16:12.724432 20834 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:16:12.744055 20834 rpc_server.cc:307] RPC server started. Bound to: 127.16.217.62:35261
I20260620 20:16:12.744057 20900 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.217.62:35261 every 8 connection(s)
I20260620 20:16:12.745527 20834 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986536687152-17252-0/minicluster-data/master-0/data/info.pb
I20260620 20:16:12.749327 20901 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:16:12.755105 17252 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu as pid 20834
I20260620 20:16:12.755360 17252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986536687152-17252-0/minicluster-data/master-0/wal/instance
I20260620 20:16:12.756173 17252 external_mini_cluster.cc:1428] Setting key 17e7c17a168df5ff2511d173272beeb1
I20260620 20:16:12.760207 20901 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 60904aed11334c78ab2c878b5701390a: Bootstrap starting.
I20260620 20:16:12.763782 20901 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 60904aed11334c78ab2c878b5701390a: Neither blocks nor log segments found. Creating new log.
I20260620 20:16:12.764796 20901 log.cc:824] T 00000000000000000000000000000000 P 60904aed11334c78ab2c878b5701390a: Log is configured to *not* fsync() on all Append() calls
I20260620 20:16:12.767688 20901 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 60904aed11334c78ab2c878b5701390a: No bootstrap required, opened a new log
Jun 20 20:16:12 dist-test-slave-2hpg krb5kdc[20821](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986572, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.217.62@KRBTEST.COM
I20260620 20:16:12.776360 20901 raft_consensus.cc:348] T 00000000000000000000000000000000 P 60904aed11334c78ab2c878b5701390a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60904aed11334c78ab2c878b5701390a" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 35261 } }
I20260620 20:16:12.776825 20901 raft_consensus.cc:374] T 00000000000000000000000000000000 P 60904aed11334c78ab2c878b5701390a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:16:12.776940 20901 raft_consensus.cc:729] T 00000000000000000000000000000000 P 60904aed11334c78ab2c878b5701390a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60904aed11334c78ab2c878b5701390a, State: Initialized, Role: FOLLOWER
I20260620 20:16:12.777611 20901 consensus_queue.cc:260] T 00000000000000000000000000000000 P 60904aed11334c78ab2c878b5701390a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60904aed11334c78ab2c878b5701390a" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 35261 } }
I20260620 20:16:12.777895 20901 raft_consensus.cc:388] T 00000000000000000000000000000000 P 60904aed11334c78ab2c878b5701390a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 20:16:12.778050 20901 raft_consensus.cc:482] T 00000000000000000000000000000000 P 60904aed11334c78ab2c878b5701390a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 20:16:12.778223 20901 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 60904aed11334c78ab2c878b5701390a [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:16:12.778978 20904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:12.758238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49010 (local address 127.16.217.62:35261)
0620 20:16:12.758979 (+ 741us) server_negotiation.cc:206] Beginning negotiation
0620 20:16:12.758989 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 20:16:12.759029 (+ 40us) server_negotiation.cc:407] Connection header received
0620 20:16:12.760683 (+ 1654us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:16:12.760705 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:16:12.761065 (+ 360us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:16:12.761366 (+ 301us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:16:12.763162 (+ 1796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:12.764719 (+ 1557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:12.766280 (+ 1561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:12.766957 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:12.767099 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:12.771693 (+ 4594us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:16:12.771700 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:16:12.771713 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:16:12.771744 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:16:12.774969 (+ 3225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:16:12.775638 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:16:12.775644 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:16:12.775650 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:16:12.775811 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:16:12.776160 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:16:12.776164 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:16:12.776166 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:16:12.776638 (+ 472us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:16:12.776894 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0620 20:16:12.777385 (+ 491us) server_negotiation.cc:299] Negotiation successful
0620 20:16:12.777739 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":548,"thread_start_us":340,"threads_started":1}
I20260620 20:16:12.781303 20901 raft_consensus.cc:504] T 00000000000000000000000000000000 P 60904aed11334c78ab2c878b5701390a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60904aed11334c78ab2c878b5701390a" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 35261 } }
I20260620 20:16:12.781901 20901 leader_election.cc:302] T 00000000000000000000000000000000 P 60904aed11334c78ab2c878b5701390a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 60904aed11334c78ab2c878b5701390a; no voters:
I20260620 20:16:12.782467 20901 leader_election.cc:288] T 00000000000000000000000000000000 P 60904aed11334c78ab2c878b5701390a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 20:16:12.782559 20906 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 60904aed11334c78ab2c878b5701390a [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:16:12.783250 20906 raft_consensus.cc:686] T 00000000000000000000000000000000 P 60904aed11334c78ab2c878b5701390a [term 1 LEADER]: Becoming Leader. State: Replica: 60904aed11334c78ab2c878b5701390a, State: Running, Role: LEADER
I20260620 20:16:12.783864 20906 consensus_queue.cc:237] T 00000000000000000000000000000000 P 60904aed11334c78ab2c878b5701390a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60904aed11334c78ab2c878b5701390a" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 35261 } }
I20260620 20:16:12.784421 20901 sys_catalog.cc:564] T 00000000000000000000000000000000 P 60904aed11334c78ab2c878b5701390a [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:16:12.786409 20908 sys_catalog.cc:455] T 00000000000000000000000000000000 P 60904aed11334c78ab2c878b5701390a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 60904aed11334c78ab2c878b5701390a. Latest consensus state: current_term: 1 leader_uuid: "60904aed11334c78ab2c878b5701390a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60904aed11334c78ab2c878b5701390a" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 35261 } } }
I20260620 20:16:12.786744 20908 sys_catalog.cc:458] T 00000000000000000000000000000000 P 60904aed11334c78ab2c878b5701390a [sys.catalog]: This master's current role is: LEADER
I20260620 20:16:12.787540 20907 sys_catalog.cc:455] T 00000000000000000000000000000000 P 60904aed11334c78ab2c878b5701390a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "60904aed11334c78ab2c878b5701390a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60904aed11334c78ab2c878b5701390a" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 35261 } } }
I20260620 20:16:12.787791 20907 sys_catalog.cc:458] T 00000000000000000000000000000000 P 60904aed11334c78ab2c878b5701390a [sys.catalog]: This master's current role is: LEADER
I20260620 20:16:12.787699 20911 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 20:16:12.797549 20911 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 20:16:12.806627 20911 catalog_manager.cc:1347] Generated new cluster ID: 0fcf9e043ae94f73b9253662da887639
I20260620 20:16:12.806747 20911 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 20:16:12.822647 20911 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 20:16:12.823966 20911 catalog_manager.cc:1504] Loading token signing keys...
I20260620 20:16:12.830967 20911 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 60904aed11334c78ab2c878b5701390a: Generated new TSK 0
I20260620 20:16:12.832046 20911 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.217.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.217.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.217.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.1.keytab.
Entry for principal kudu/127.16.217.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.1.keytab.
I20260620 20:16:12.875103 17252 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
/tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986536687152-17252-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986536687152-17252-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986536687152-17252-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986536687152-17252-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.217.1:0
--local_ip_for_outbound_sockets=127.16.217.1
--webserver_interface=127.16.217.1
--webserver_port=0
--tserver_master_addrs=127.16.217.62:35261
--keytab_file=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.1.keytab
--principal=kudu/127.16.217.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.217.20:43717
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986536687152-17252-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986536687152-17252-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986536687152-17252-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:16:13.198902 20927 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:16:13.199159 20927 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:16:13.199328 20927 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:16:13.206485 20927 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:16:13.206606 20927 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:16:13.206789 20927 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.217.1
I20260620 20:16:13.216214 20927 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.217.20:43717
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986536687152-17252-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986536687152-17252-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.1.keytab
--principal=kudu/127.16.217.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.217.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986536687152-17252-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.217.1
--webserver_port=0
--tserver_master_addrs=127.16.217.62:35261
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.217.1
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986536687152-17252-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:16:13.217406 20927 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:16:13.218829 20927 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:16:13.231691 20933 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:16:13.231745 20932 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:16:13.231769 20935 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:16:13.232402 20927 server_base.cc:1034] running on GCE node
I20260620 20:16:13.233060 20927 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:16:13.234037 20927 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:16:13.235245 20927 hybrid_clock.cc:648] HybridClock initialized: now 1781986573235199 us; error 46 us; skew 500 ppm
I20260620 20:16:13.235546 20927 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:16:13 dist-test-slave-2hpg krb5kdc[20821](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986573, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.217.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:16:13.238631 20927 init.cc:377] Logged in from keytab as kudu/127.16.217.1@KRBTEST.COM (short username kudu)
I20260620 20:16:13.239934 20927 webserver.cc:460] Webserver started at http://127.16.217.1:44447/ using document root <none> and password file <none>
I20260620 20:16:13.240528 20927 fs_manager.cc:359] Metadata directory not provided
I20260620 20:16:13.240618 20927 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:16:13.240850 20927 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:16:13.243547 20927 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986536687152-17252-0/minicluster-data/ts-0/data/instance:
uuid: "37e01689f07344ce854a05c22f6d1f29"
format_stamp: "Formatted at 2026-06-20 20:16:13 on dist-test-slave-2hpg"
server_key: "f7f8fd9d9f5c56ca37e72437f7d7545e"
server_key_iv: "a36e7793407271314cee319363f0300b"
server_key_version: "encryptionkey@0"
I20260620 20:16:13.244217 20927 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986536687152-17252-0/minicluster-data/ts-0/wal/instance:
uuid: "37e01689f07344ce854a05c22f6d1f29"
format_stamp: "Formatted at 2026-06-20 20:16:13 on dist-test-slave-2hpg"
server_key: "f7f8fd9d9f5c56ca37e72437f7d7545e"
server_key_iv: "a36e7793407271314cee319363f0300b"
server_key_version: "encryptionkey@0"
I20260620 20:16:13.248962 20927 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260620 20:16:13.252188 20942 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:16:13.253434 20927 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 20:16:13.253554 20927 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986536687152-17252-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986536687152-17252-0/minicluster-data/ts-0/wal
uuid: "37e01689f07344ce854a05c22f6d1f29"
format_stamp: "Formatted at 2026-06-20 20:16:13 on dist-test-slave-2hpg"
server_key: "f7f8fd9d9f5c56ca37e72437f7d7545e"
server_key_iv: "a36e7793407271314cee319363f0300b"
server_key_version: "encryptionkey@0"
I20260620 20:16:13.253705 20927 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986536687152-17252-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986536687152-17252-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986536687152-17252-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:16:13.275367 20927 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:16:13.280030 20927 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:16:13.280295 20927 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:16:13.281206 20927 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:16:13.282632 20927 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:16:13.282716 20927 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:16:13.282819 20927 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:16:13.282869 20927 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:16:13.317219 20927 rpc_server.cc:307] RPC server started. Bound to: 127.16.217.1:42593
I20260620 20:16:13.317229 21054 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.217.1:42593 every 8 connection(s)
I20260620 20:16:13.318637 20927 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986536687152-17252-0/minicluster-data/ts-0/data/info.pb
I20260620 20:16:13.327586 17252 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu as pid 20927
I20260620 20:16:13.327853 17252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986536687152-17252-0/minicluster-data/ts-0/wal/instance
I20260620 20:16:13.328492 17252 external_mini_cluster.cc:1428] Setting key ddd2d7b7b5767ce01dcd0e1dddfd7e74
Jun 20 20:16:13 dist-test-slave-2hpg krb5kdc[20821](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986573, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.217.1@KRBTEST.COM for kudu/127.16.217.62@KRBTEST.COM
I20260620 20:16:13.339679 21057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:13.321207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.217.1:60499 (local address 127.16.217.62:35261)
0620 20:16:13.321683 (+ 476us) server_negotiation.cc:206] Beginning negotiation
0620 20:16:13.321690 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:16:13.322752 (+ 1062us) server_negotiation.cc:407] Connection header received
0620 20:16:13.324548 (+ 1796us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:16:13.324553 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:16:13.324623 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:16:13.324742 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:16:13.326312 (+ 1570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:13.327045 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:13.328800 (+ 1755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:13.329354 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:13.329491 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:13.333432 (+ 3941us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:16:13.333438 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:16:13.333444 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:16:13.333484 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:16:13.336513 (+ 3029us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:16:13.337263 (+ 750us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:16:13.337269 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:16:13.337273 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:16:13.337399 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:16:13.337847 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:16:13.337852 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:16:13.337856 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:16:13.338177 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:16:13.338346 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0620 20:16:13.339038 (+ 692us) server_negotiation.cc:299] Negotiation successful
0620 20:16:13.339266 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":348,"thread_start_us":215,"threads_started":1}
I20260620 20:16:13.340649 21058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:13.321819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:35261 (local address 127.16.217.1:60499)
0620 20:16:13.322585 (+ 766us) negotiation.cc:107] Waiting for socket to connect
0620 20:16:13.322623 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 20:16:13.324293 (+ 1670us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:16:13.324947 (+ 654us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:16:13.324954 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:16:13.325431 (+ 477us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:16:13.326155 (+ 724us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:16:13.326176 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:16:13.327215 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:16:13.327219 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:16:13.328623 (+ 1404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:16:13.328635 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:16:13.329569 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:16:13.329577 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:16:13.329854 (+ 277us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:13.330579 (+ 725us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:16:13.330624 (+ 45us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:16:13.333165 (+ 2541us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:16:13.336656 (+ 3491us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:16:13.336665 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:16:13.336681 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:16:13.337125 (+ 444us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:16:13.337517 (+ 392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:16:13.337523 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:16:13.337526 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:16:13.337717 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:16:13.338319 (+ 602us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:16:13.338327 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:16:13.338658 (+ 331us) client_negotiation.cc:769] Sending connection context
0620 20:16:13.338945 (+ 287us) client_negotiation.cc:240] Negotiation successful
0620 20:16:13.339447 (+ 502us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":540,"thread_start_us":299,"threads_started":1}
I20260620 20:16:13.342272 21055 heartbeater.cc:347] Connected to a master server at 127.16.217.62:35261
I20260620 20:16:13.342563 21055 heartbeater.cc:464] Registering TS with master...
I20260620 20:16:13.343216 21055 heartbeater.cc:511] Master 127.16.217.62:35261 requested a full tablet report, sending...
I20260620 20:16:13.345152 20866 ts_manager.cc:194] Registered new tserver with Master: 37e01689f07344ce854a05c22f6d1f29 (127.16.217.1:42593)
WARNING: no policy specified for kudu/127.16.217.2@KRBTEST.COM; defaulting to no policy
I20260620 20:16:13.346599 20866 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.217.1@KRBTEST.COM'} at 127.16.217.1:60499
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.217.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.217.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.2.keytab.
Entry for principal kudu/127.16.217.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.2.keytab.
I20260620 20:16:13.370707 17252 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
/tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986536687152-17252-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986536687152-17252-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986536687152-17252-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986536687152-17252-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.217.2:0
--local_ip_for_outbound_sockets=127.16.217.2
--webserver_interface=127.16.217.2
--webserver_port=0
--tserver_master_addrs=127.16.217.62:35261
--keytab_file=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.2.keytab
--principal=kudu/127.16.217.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.217.20:43717
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986536687152-17252-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986536687152-17252-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986536687152-17252-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:16:13.691560 21061 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:16:13.691843 21061 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:16:13.692022 21061 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:16:13.699436 21061 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:16:13.699548 21061 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:16:13.699862 21061 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.217.2
I20260620 20:16:13.709417 21061 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.217.20:43717
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986536687152-17252-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986536687152-17252-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.2.keytab
--principal=kudu/127.16.217.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.217.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986536687152-17252-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.217.2
--webserver_port=0
--tserver_master_addrs=127.16.217.62:35261
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.217.2
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986536687152-17252-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:16:13.710657 21061 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:16:13.711964 21061 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:16:13.725414 21066 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:16:13.725370 21067 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:16:13.725456 21069 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:16:13.726259 21061 server_base.cc:1034] running on GCE node
I20260620 20:16:13.726881 21061 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:16:13.727811 21061 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:16:13.729014 21061 hybrid_clock.cc:648] HybridClock initialized: now 1781986573728961 us; error 36 us; skew 500 ppm
I20260620 20:16:13.729346 21061 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:16:13 dist-test-slave-2hpg krb5kdc[20821](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986573, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.217.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:16:13.732564 21061 init.cc:377] Logged in from keytab as kudu/127.16.217.2@KRBTEST.COM (short username kudu)
I20260620 20:16:13.734030 21061 webserver.cc:460] Webserver started at http://127.16.217.2:37763/ using document root <none> and password file <none>
I20260620 20:16:13.734645 21061 fs_manager.cc:359] Metadata directory not provided
I20260620 20:16:13.734760 21061 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:16:13.735031 21061 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:16:13.737951 21061 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986536687152-17252-0/minicluster-data/ts-1/data/instance:
uuid: "eb298c95f6af47998656322b42143c87"
format_stamp: "Formatted at 2026-06-20 20:16:13 on dist-test-slave-2hpg"
server_key: "cbcb8d6d7c53d02d15ae5c9680eee819"
server_key_iv: "eaf1bef88c34dc6e5a6eb53b96513212"
server_key_version: "encryptionkey@0"
I20260620 20:16:13.738718 21061 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986536687152-17252-0/minicluster-data/ts-1/wal/instance:
uuid: "eb298c95f6af47998656322b42143c87"
format_stamp: "Formatted at 2026-06-20 20:16:13 on dist-test-slave-2hpg"
server_key: "cbcb8d6d7c53d02d15ae5c9680eee819"
server_key_iv: "eaf1bef88c34dc6e5a6eb53b96513212"
server_key_version: "encryptionkey@0"
I20260620 20:16:13.743795 21061 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 20:16:13.747205 21076 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:16:13.748582 21061 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 20:16:13.748740 21061 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986536687152-17252-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986536687152-17252-0/minicluster-data/ts-1/wal
uuid: "eb298c95f6af47998656322b42143c87"
format_stamp: "Formatted at 2026-06-20 20:16:13 on dist-test-slave-2hpg"
server_key: "cbcb8d6d7c53d02d15ae5c9680eee819"
server_key_iv: "eaf1bef88c34dc6e5a6eb53b96513212"
server_key_version: "encryptionkey@0"
I20260620 20:16:13.748929 21061 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986536687152-17252-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986536687152-17252-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986536687152-17252-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:16:13.775570 21061 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:16:13.780056 21061 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:16:13.780350 21061 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:16:13.781312 21061 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:16:13.782688 21061 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:16:13.782792 21061 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:16:13.782927 21061 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:16:13.782994 21061 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:16:13.816256 21061 rpc_server.cc:307] RPC server started. Bound to: 127.16.217.2:33179
I20260620 20:16:13.816339 21188 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.217.2:33179 every 8 connection(s)
I20260620 20:16:13.817519 21061 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986536687152-17252-0/minicluster-data/ts-1/data/info.pb
I20260620 20:16:13.818476 17252 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu as pid 21061
I20260620 20:16:13.818676 17252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986536687152-17252-0/minicluster-data/ts-1/wal/instance
I20260620 20:16:13.819124 17252 external_mini_cluster.cc:1428] Setting key e1e1a7475679fa073f8476bcaac4c233
Jun 20 20:16:13 dist-test-slave-2hpg krb5kdc[20821](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986573, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.217.2@KRBTEST.COM for kudu/127.16.217.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.16.217.3@KRBTEST.COM; defaulting to no policy
I20260620 20:16:13.840287 21057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:13.820430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.217.2:54755 (local address 127.16.217.62:35261)
0620 20:16:13.820617 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0620 20:16:13.820624 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:16:13.822045 (+ 1421us) server_negotiation.cc:407] Connection header received
0620 20:16:13.825012 (+ 2967us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:16:13.825016 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:16:13.825081 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:16:13.825168 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:16:13.827302 (+ 2134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:13.828421 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:13.830018 (+ 1597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:13.830771 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:13.830918 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:13.834616 (+ 3698us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:16:13.834626 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:16:13.834633 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:16:13.834676 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:16:13.837350 (+ 2674us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:16:13.838123 (+ 773us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:16:13.838128 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:16:13.838136 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:16:13.838258 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:16:13.838665 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:16:13.838670 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:16:13.838674 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:16:13.838986 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:16:13.839128 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0620 20:16:13.839867 (+ 739us) server_negotiation.cc:299] Negotiation successful
0620 20:16:13.840037 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.217.3@KRBTEST.COM" created.
I20260620 20:16:13.841436 21191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:13.820989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:35261 (local address 127.16.217.2:54755)
0620 20:16:13.821852 (+ 863us) negotiation.cc:107] Waiting for socket to connect
0620 20:16:13.821940 (+ 88us) client_negotiation.cc:174] Beginning negotiation
0620 20:16:13.824703 (+ 2763us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:16:13.825483 (+ 780us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:16:13.825498 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:16:13.826323 (+ 825us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:16:13.827129 (+ 806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:16:13.827156 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:16:13.828574 (+ 1418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:16:13.828577 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:16:13.829827 (+ 1250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:16:13.829843 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:16:13.831005 (+ 1162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:16:13.831012 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:16:13.831316 (+ 304us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:13.832006 (+ 690us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:16:13.832027 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:16:13.834301 (+ 2274us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:16:13.837500 (+ 3199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:16:13.837508 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:16:13.837524 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:16:13.838007 (+ 483us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:16:13.838370 (+ 363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:16:13.838375 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:16:13.838378 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:16:13.838552 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:16:13.839104 (+ 552us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:16:13.839112 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:16:13.839448 (+ 336us) client_negotiation.cc:769] Sending connection context
0620 20:16:13.839756 (+ 308us) client_negotiation.cc:240] Negotiation successful
0620 20:16:13.840132 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":563,"thread_start_us":317,"threads_started":1}
I20260620 20:16:13.842897 21189 heartbeater.cc:347] Connected to a master server at 127.16.217.62:35261
I20260620 20:16:13.843216 21189 heartbeater.cc:464] Registering TS with master...
I20260620 20:16:13.844060 21189 heartbeater.cc:511] Master 127.16.217.62:35261 requested a full tablet report, sending...
I20260620 20:16:13.845597 20866 ts_manager.cc:194] Registered new tserver with Master: eb298c95f6af47998656322b42143c87 (127.16.217.2:33179)
I20260620 20:16:13.846834 20866 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.217.2@KRBTEST.COM'} at 127.16.217.2:54755
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.217.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.3.keytab.
Entry for principal kudu/127.16.217.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.3.keytab.
I20260620 20:16:13.864497 17252 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
/tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986536687152-17252-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986536687152-17252-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986536687152-17252-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986536687152-17252-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.217.3:0
--local_ip_for_outbound_sockets=127.16.217.3
--webserver_interface=127.16.217.3
--webserver_port=0
--tserver_master_addrs=127.16.217.62:35261
--keytab_file=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.3.keytab
--principal=kudu/127.16.217.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.217.20:43717
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986536687152-17252-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986536687152-17252-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986536687152-17252-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:16:14.177631 21194 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:16:14.177932 21194 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:16:14.178028 21194 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:16:14.185232 21194 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:16:14.185336 21194 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:16:14.185495 21194 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.217.3
I20260620 20:16:14.195188 21194 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.217.20:43717
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986536687152-17252-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986536687152-17252-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.3.keytab
--principal=kudu/127.16.217.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.217.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986536687152-17252-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.217.3
--webserver_port=0
--tserver_master_addrs=127.16.217.62:35261
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.217.3
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986536687152-17252-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:16:14.196370 21194 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:16:14.197588 21194 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:16:14.210352 21202 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:16:14.210404 21199 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:16:14.210429 21200 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:16:14.211109 21194 server_base.cc:1034] running on GCE node
I20260620 20:16:14.211658 21194 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:16:14.212505 21194 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:16:14.213688 21194 hybrid_clock.cc:648] HybridClock initialized: now 1781986574213617 us; error 69 us; skew 500 ppm
I20260620 20:16:14.213992 21194 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:16:14 dist-test-slave-2hpg krb5kdc[20821](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986574, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.217.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:16:14.216964 21194 init.cc:377] Logged in from keytab as kudu/127.16.217.3@KRBTEST.COM (short username kudu)
I20260620 20:16:14.218359 21194 webserver.cc:460] Webserver started at http://127.16.217.3:40597/ using document root <none> and password file <none>
I20260620 20:16:14.218992 21194 fs_manager.cc:359] Metadata directory not provided
I20260620 20:16:14.219116 21194 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:16:14.219416 21194 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:16:14.222227 21194 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986536687152-17252-0/minicluster-data/ts-2/data/instance:
uuid: "866ca5d59ba64a2f8be1d052c054bf01"
format_stamp: "Formatted at 2026-06-20 20:16:14 on dist-test-slave-2hpg"
server_key: "fbcf5dfd567d4a03adc4152bec81439e"
server_key_iv: "3c9c5cc559621634a2f0e64d955215b2"
server_key_version: "encryptionkey@0"
I20260620 20:16:14.222972 21194 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986536687152-17252-0/minicluster-data/ts-2/wal/instance:
uuid: "866ca5d59ba64a2f8be1d052c054bf01"
format_stamp: "Formatted at 2026-06-20 20:16:14 on dist-test-slave-2hpg"
server_key: "fbcf5dfd567d4a03adc4152bec81439e"
server_key_iv: "3c9c5cc559621634a2f0e64d955215b2"
server_key_version: "encryptionkey@0"
I20260620 20:16:14.227756 21194 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260620 20:16:14.231038 21209 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:16:14.232328 21194 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260620 20:16:14.232491 21194 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986536687152-17252-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986536687152-17252-0/minicluster-data/ts-2/wal
uuid: "866ca5d59ba64a2f8be1d052c054bf01"
format_stamp: "Formatted at 2026-06-20 20:16:14 on dist-test-slave-2hpg"
server_key: "fbcf5dfd567d4a03adc4152bec81439e"
server_key_iv: "3c9c5cc559621634a2f0e64d955215b2"
server_key_version: "encryptionkey@0"
I20260620 20:16:14.232683 21194 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986536687152-17252-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986536687152-17252-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986536687152-17252-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:16:14.256059 21194 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:16:14.260120 21194 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:16:14.260407 21194 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:16:14.261291 21194 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:16:14.262733 21194 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:16:14.262805 21194 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:16:14.262885 21194 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:16:14.262928 21194 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:16:14.294179 21194 rpc_server.cc:307] RPC server started. Bound to: 127.16.217.3:40861
I20260620 20:16:14.294269 21321 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.217.3:40861 every 8 connection(s)
I20260620 20:16:14.295459 21194 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986536687152-17252-0/minicluster-data/ts-2/data/info.pb
I20260620 20:16:14.301501 17252 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu as pid 21194
I20260620 20:16:14.301724 17252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986536687152-17252-0/minicluster-data/ts-2/wal/instance
I20260620 20:16:14.302304 17252 external_mini_cluster.cc:1428] Setting key d1e577d77c57602987ee3f01c6ab69b4
Jun 20 20:16:14 dist-test-slave-2hpg krb5kdc[20821](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986574, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.217.3@KRBTEST.COM for kudu/127.16.217.62@KRBTEST.COM
I20260620 20:16:14.315788 21057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:14.297949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.217.3:50317 (local address 127.16.217.62:35261)
0620 20:16:14.298155 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0620 20:16:14.298160 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:16:14.299458 (+ 1298us) server_negotiation.cc:407] Connection header received
0620 20:16:14.301440 (+ 1982us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:16:14.301447 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:16:14.301593 (+ 146us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:16:14.301727 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:16:14.303789 (+ 2062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:14.304646 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:14.306365 (+ 1719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:14.306778 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:14.306905 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:14.310310 (+ 3405us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:16:14.310315 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:16:14.310319 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:16:14.310349 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:16:14.312980 (+ 2631us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:16:14.313629 (+ 649us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:16:14.313632 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:16:14.313635 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:16:14.313752 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:16:14.314223 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:16:14.314226 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:16:14.314228 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:16:14.314503 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:16:14.314608 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0620 20:16:14.315283 (+ 675us) server_negotiation.cc:299] Negotiation successful
0620 20:16:14.315455 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260620 20:16:14.317023 21324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:14.298529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:35261 (local address 127.16.217.3:50317)
0620 20:16:14.299284 (+ 755us) negotiation.cc:107] Waiting for socket to connect
0620 20:16:14.299326 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 20:16:14.301137 (+ 1811us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:16:14.302094 (+ 957us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:16:14.302106 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:16:14.302858 (+ 752us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:16:14.303618 (+ 760us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:16:14.303647 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:16:14.304835 (+ 1188us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:16:14.304840 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:16:14.306225 (+ 1385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:16:14.306237 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:16:14.306992 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:16:14.307000 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:16:14.307256 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:14.307917 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:16:14.307939 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:16:14.310105 (+ 2166us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:16:14.313093 (+ 2988us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:16:14.313099 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:16:14.313112 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:16:14.313529 (+ 417us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:16:14.313933 (+ 404us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:16:14.313937 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:16:14.313940 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:16:14.314123 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:16:14.314609 (+ 486us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:16:14.314614 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:16:14.314934 (+ 320us) client_negotiation.cc:769] Sending connection context
0620 20:16:14.315151 (+ 217us) client_negotiation.cc:240] Negotiation successful
0620 20:16:14.315629 (+ 478us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":500,"thread_start_us":292,"threads_started":1}
I20260620 20:16:14.318403 21322 heartbeater.cc:347] Connected to a master server at 127.16.217.62:35261
I20260620 20:16:14.318701 21322 heartbeater.cc:464] Registering TS with master...
I20260620 20:16:14.319429 21322 heartbeater.cc:511] Master 127.16.217.62:35261 requested a full tablet report, sending...
I20260620 20:16:14.320912 20866 ts_manager.cc:194] Registered new tserver with Master: 866ca5d59ba64a2f8be1d052c054bf01 (127.16.217.3:40861)
I20260620 20:16:14.321976 20866 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.217.3@KRBTEST.COM'} at 127.16.217.3:50317
I20260620 20:16:14.329515 17252 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 20:16:14.349332 21055 heartbeater.cc:503] Master 127.16.217.62:35261 was elected leader, sending a full tablet report...
Jun 20 20:16:14 dist-test-slave-2hpg krb5kdc[20821](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986574, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 20:16:14 dist-test-slave-2hpg krb5kdc[20821](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986574, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.217.1@KRBTEST.COM
I20260620 20:16:14.376523 21329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:14.361940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54054 (local address 127.16.217.1:42593)
0620 20:16:14.362479 (+ 539us) server_negotiation.cc:206] Beginning negotiation
0620 20:16:14.362486 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:16:14.362510 (+ 24us) server_negotiation.cc:407] Connection header received
0620 20:16:14.362570 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:16:14.362576 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:16:14.362781 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:16:14.362927 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:16:14.364216 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:14.365186 (+ 970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:14.366462 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:14.366995 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:14.367088 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:14.370994 (+ 3906us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:16:14.371007 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:16:14.371022 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:16:14.371061 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:16:14.373985 (+ 2924us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:16:14.374553 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:16:14.374560 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:16:14.374562 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:16:14.374686 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:16:14.375064 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:16:14.375067 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:16:14.375069 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:16:14.375412 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:16:14.375633 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0620 20:16:14.375923 (+ 290us) server_negotiation.cc:299] Negotiation successful
0620 20:16:14.376147 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":408,"thread_start_us":260,"threads_started":1}
Jun 20 20:16:14 dist-test-slave-2hpg krb5kdc[20821](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986574, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.217.62@KRBTEST.COM
I20260620 20:16:14.395409 21057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:14.380935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49018 (local address 127.16.217.62:35261)
0620 20:16:14.381168 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0620 20:16:14.381174 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:16:14.381321 (+ 147us) server_negotiation.cc:407] Connection header received
0620 20:16:14.381540 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:16:14.381544 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:16:14.381607 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:16:14.381713 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:16:14.383049 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:14.383849 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:14.385300 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:14.385758 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:14.385948 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:14.389963 (+ 4015us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:16:14.389969 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:16:14.389972 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:16:14.390007 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:16:14.392628 (+ 2621us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:16:14.393317 (+ 689us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:16:14.393325 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:16:14.393329 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:16:14.393472 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:16:14.393895 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:16:14.393902 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:16:14.393905 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:16:14.394251 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:16:14.394512 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0620 20:16:14.394869 (+ 357us) server_negotiation.cc:299] Negotiation successful
0620 20:16:14.395087 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
W20260620 20:16:14.395998 20866 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49018
I20260620 20:16:14.397136 17252 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu pid 20927
I20260620 20:16:14.408728 21329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:14.398124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54064 (local address 127.16.217.1:42593)
0620 20:16:14.398313 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0620 20:16:14.398319 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:16:14.398412 (+ 93us) server_negotiation.cc:407] Connection header received
0620 20:16:14.398544 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:16:14.398548 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:16:14.398634 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:16:14.398706 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:16:14.399807 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:14.400464 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:14.401596 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:14.402077 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:14.402175 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:14.404157 (+ 1982us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:16:14.404165 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:16:14.404173 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:16:14.404222 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:16:14.406708 (+ 2486us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:16:14.407248 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:16:14.407251 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:16:14.407254 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:16:14.407365 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:16:14.407639 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:16:14.407642 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:16:14.407644 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:16:14.407888 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:16:14.408045 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0620 20:16:14.408270 (+ 225us) server_negotiation.cc:299] Negotiation successful
0620 20:16:14.408414 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 20:16:14.409329 21050 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:16:14.490813 17252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu with pid 20927
I20260620 20:16:14.506589 17252 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu pid 21061
Jun 20 20:16:14 dist-test-slave-2hpg krb5kdc[20821](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986574, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.217.2@KRBTEST.COM
I20260620 20:16:14.521605 21334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:14.507453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35152 (local address 127.16.217.2:33179)
0620 20:16:14.507985 (+ 532us) server_negotiation.cc:206] Beginning negotiation
0620 20:16:14.507992 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:16:14.508017 (+ 25us) server_negotiation.cc:407] Connection header received
0620 20:16:14.508077 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:16:14.508083 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:16:14.508293 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:16:14.508418 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:16:14.509506 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:14.510521 (+ 1015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:14.511721 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:14.512221 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:14.512309 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:14.516371 (+ 4062us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:16:14.516382 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:16:14.516396 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:16:14.516432 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:16:14.519228 (+ 2796us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:16:14.519782 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:16:14.519829 (+ 47us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:16:14.519833 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:16:14.519987 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:16:14.520297 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:16:14.520302 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:16:14.520305 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:16:14.520633 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:16:14.520872 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0620 20:16:14.521123 (+ 251us) server_negotiation.cc:299] Negotiation successful
0620 20:16:14.521315 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":387,"thread_start_us":264,"threads_started":1}
I20260620 20:16:14.522912 21184 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:16:14.619171 17252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu with pid 21061
I20260620 20:16:14.635437 17252 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu pid 21194
Jun 20 20:16:14 dist-test-slave-2hpg krb5kdc[20821](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986574, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.217.3@KRBTEST.COM
I20260620 20:16:14.653172 21336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:14.636596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34104 (local address 127.16.217.3:40861)
0620 20:16:14.637413 (+ 817us) server_negotiation.cc:206] Beginning negotiation
0620 20:16:14.637420 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:16:14.637450 (+ 30us) server_negotiation.cc:407] Connection header received
0620 20:16:14.637508 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:16:14.637515 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:16:14.637730 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:16:14.637970 (+ 240us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:16:14.639194 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:14.640231 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:14.641747 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:14.642510 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:14.642668 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:14.647532 (+ 4864us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:16:14.647543 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:16:14.647557 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:16:14.647588 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:16:14.650690 (+ 3102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:16:14.651304 (+ 614us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:16:14.651310 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:16:14.651313 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:16:14.651427 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:16:14.651804 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:16:14.651807 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:16:14.651809 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:16:14.652133 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:16:14.652354 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0620 20:16:14.652663 (+ 309us) server_negotiation.cc:299] Negotiation successful
0620 20:16:14.652892 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":674,"thread_start_us":367,"threads_started":1}
I20260620 20:16:14.654614 21317 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:16:14.736574 17252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu with pid 21194
I20260620 20:16:14.751814 17252 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu pid 20834
I20260620 20:16:14.752894 20896 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:16:14.852900 17252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu with pid 20834
2026-06-20T20:16:14Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (3829 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986536687152-17252-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 20:16:14 dist-test-slave-2hpg krb5kdc[21341](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 20:16:14 dist-test-slave-2hpg krb5kdc[21341](info): set up 2 sockets
Jun 20 20:16:14 dist-test-slave-2hpg krb5kdc[21341](info): commencing operation
krb5kdc: starting...
W20260620 20:16:15.942651 17252 mini_kdc.cc:121] Time spent starting KDC: real 1.047s user 0.000s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986536687152-17252-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986536687152-17252-0/krb5kdc/test-user.keytab.
Jun 20 20:16:16 dist-test-slave-2hpg krb5kdc[21341](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986576, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T20:16:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T20:16:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.217.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.217.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.217.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.62.keytab.
Entry for principal kudu/127.16.217.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.62.keytab.
I20260620 20:16:16.138962 17252 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
/tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986536687152-17252-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986536687152-17252-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986536687152-17252-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986536687152-17252-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.217.62:46229
--webserver_interface=127.16.217.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.62.keytab
--principal=kudu/127.16.217.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.217.20:33849
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.217.62:46229
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986536687152-17252-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986536687152-17252-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986536687152-17252-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:16:16.457149 21354 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:16:16.457450 21354 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:16:16.457660 21354 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:16:16.465279 21354 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:16:16.465391 21354 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:16:16.465492 21354 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:16:16.465600 21354 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:16:16.465664 21354 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:16:16.475806 21354 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.217.20:33849
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986536687152-17252-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986536687152-17252-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.217.62:46229
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.62.keytab
--principal=kudu/127.16.217.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.217.62:46229
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986536687152-17252-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.217.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986536687152-17252-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:16:16.477630 21354 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:16:16.478978 21354 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:16:16.490592 21360 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:16:16.490648 21362 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:16:16.491104 21359 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:16:16.491537 21354 server_base.cc:1034] running on GCE node
I20260620 20:16:16.492300 21354 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:16:16.493638 21354 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:16:16.494912 21354 hybrid_clock.cc:648] HybridClock initialized: now 1781986576494851 us; error 77 us; skew 500 ppm
I20260620 20:16:16.495254 21354 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:16:16 dist-test-slave-2hpg krb5kdc[21341](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986576, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.217.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:16:16.498353 21354 init.cc:377] Logged in from keytab as kudu/127.16.217.62@KRBTEST.COM (short username kudu)
I20260620 20:16:16.499478 21354 fs_manager.cc:359] Metadata directory not provided
I20260620 20:16:16.499565 21354 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:16:16.499820 21354 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:16:16.502722 21354 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986536687152-17252-0/minicluster-data/master-0/data/instance:
uuid: "82d02b2d07714d6791a4888da6376c37"
format_stamp: "Formatted at 2026-06-20 20:16:16 on dist-test-slave-2hpg"
server_key: "c635d059c26f7b115bb3aa9f7b7ea96f"
server_key_iv: "0cd4679f590643a2f2e1b794a13dbcba"
server_key_version: "encryptionkey@0"
I20260620 20:16:16.503443 21354 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986536687152-17252-0/minicluster-data/master-0/wal/instance:
uuid: "82d02b2d07714d6791a4888da6376c37"
format_stamp: "Formatted at 2026-06-20 20:16:16 on dist-test-slave-2hpg"
server_key: "c635d059c26f7b115bb3aa9f7b7ea96f"
server_key_iv: "0cd4679f590643a2f2e1b794a13dbcba"
server_key_version: "encryptionkey@0"
I20260620 20:16:16.508409 21354 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 20:16:16.511776 21368 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:16:16.513082 21354 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 20:16:16.513258 21354 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986536687152-17252-0/minicluster-data/master-0/data,/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986536687152-17252-0/minicluster-data/master-0/wal
uuid: "82d02b2d07714d6791a4888da6376c37"
format_stamp: "Formatted at 2026-06-20 20:16:16 on dist-test-slave-2hpg"
server_key: "c635d059c26f7b115bb3aa9f7b7ea96f"
server_key_iv: "0cd4679f590643a2f2e1b794a13dbcba"
server_key_version: "encryptionkey@0"
I20260620 20:16:16.513463 21354 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986536687152-17252-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986536687152-17252-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986536687152-17252-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:16:16.528136 21354 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:16:16.532200 21354 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:16:16.532474 21354 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:16:16.550712 21354 rpc_server.cc:307] RPC server started. Bound to: 127.16.217.62:46229
I20260620 20:16:16.550737 21419 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.217.62:46229 every 8 connection(s)
I20260620 20:16:16.551600 21354 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986536687152-17252-0/minicluster-data/master-0/data/info.pb
I20260620 20:16:16.555101 21420 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:16:16.557706 17252 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu as pid 21354
I20260620 20:16:16.557960 17252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986536687152-17252-0/minicluster-data/master-0/wal/instance
I20260620 20:16:16.558727 17252 external_mini_cluster.cc:1428] Setting key ec1ffa73e845513b719980b551548345
I20260620 20:16:16.568807 21420 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 82d02b2d07714d6791a4888da6376c37: Bootstrap starting.
I20260620 20:16:16.572302 21420 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 82d02b2d07714d6791a4888da6376c37: Neither blocks nor log segments found. Creating new log.
I20260620 20:16:16.573426 21420 log.cc:824] T 00000000000000000000000000000000 P 82d02b2d07714d6791a4888da6376c37: Log is configured to *not* fsync() on all Append() calls
Jun 20 20:16:16 dist-test-slave-2hpg krb5kdc[21341](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986576, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.217.62@KRBTEST.COM
I20260620 20:16:16.576699 21420 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 82d02b2d07714d6791a4888da6376c37: No bootstrap required, opened a new log
I20260620 20:16:16.583133 21423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:16.560683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32824 (local address 127.16.217.62:46229)
0620 20:16:16.561461 (+ 778us) server_negotiation.cc:206] Beginning negotiation
0620 20:16:16.561488 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0620 20:16:16.561536 (+ 48us) server_negotiation.cc:407] Connection header received
0620 20:16:16.563914 (+ 2378us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:16:16.563941 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:16:16.564407 (+ 466us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:16:16.564828 (+ 421us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:16:16.566190 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:16.568122 (+ 1932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:16.569412 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:16.570450 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:16.570635 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:16.574728 (+ 4093us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:16:16.574741 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:16:16.574762 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:16:16.574861 (+ 99us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:16:16.578673 (+ 3812us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:16:16.579298 (+ 625us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:16:16.579306 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:16:16.579314 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:16:16.579524 (+ 210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:16:16.579878 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:16:16.579885 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:16:16.579890 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:16:16.580592 (+ 702us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:16:16.580876 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0620 20:16:16.581233 (+ 357us) server_negotiation.cc:299] Negotiation successful
0620 20:16:16.581648 (+ 415us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":547,"thread_start_us":336,"threads_started":1}
I20260620 20:16:16.586788 21420 raft_consensus.cc:348] T 00000000000000000000000000000000 P 82d02b2d07714d6791a4888da6376c37 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82d02b2d07714d6791a4888da6376c37" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 46229 } }
I20260620 20:16:16.587091 21420 raft_consensus.cc:374] T 00000000000000000000000000000000 P 82d02b2d07714d6791a4888da6376c37 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:16:16.587162 21420 raft_consensus.cc:729] T 00000000000000000000000000000000 P 82d02b2d07714d6791a4888da6376c37 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82d02b2d07714d6791a4888da6376c37, State: Initialized, Role: FOLLOWER
I20260620 20:16:16.587617 21420 consensus_queue.cc:260] T 00000000000000000000000000000000 P 82d02b2d07714d6791a4888da6376c37 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82d02b2d07714d6791a4888da6376c37" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 46229 } }
I20260620 20:16:16.587839 21420 raft_consensus.cc:388] T 00000000000000000000000000000000 P 82d02b2d07714d6791a4888da6376c37 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 20:16:16.587983 21420 raft_consensus.cc:482] T 00000000000000000000000000000000 P 82d02b2d07714d6791a4888da6376c37 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 20:16:16.588164 21420 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 82d02b2d07714d6791a4888da6376c37 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:16:16.590394 21420 raft_consensus.cc:504] T 00000000000000000000000000000000 P 82d02b2d07714d6791a4888da6376c37 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82d02b2d07714d6791a4888da6376c37" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 46229 } }
I20260620 20:16:16.590863 21420 leader_election.cc:302] T 00000000000000000000000000000000 P 82d02b2d07714d6791a4888da6376c37 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 82d02b2d07714d6791a4888da6376c37; no voters:
I20260620 20:16:16.591393 21420 leader_election.cc:288] T 00000000000000000000000000000000 P 82d02b2d07714d6791a4888da6376c37 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 20:16:16.591553 21425 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 82d02b2d07714d6791a4888da6376c37 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:16:16.592203 21425 raft_consensus.cc:686] T 00000000000000000000000000000000 P 82d02b2d07714d6791a4888da6376c37 [term 1 LEADER]: Becoming Leader. State: Replica: 82d02b2d07714d6791a4888da6376c37, State: Running, Role: LEADER
I20260620 20:16:16.592667 21425 consensus_queue.cc:237] T 00000000000000000000000000000000 P 82d02b2d07714d6791a4888da6376c37 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82d02b2d07714d6791a4888da6376c37" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 46229 } }
I20260620 20:16:16.593636 21420 sys_catalog.cc:564] T 00000000000000000000000000000000 P 82d02b2d07714d6791a4888da6376c37 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:16:16.595688 21427 sys_catalog.cc:455] T 00000000000000000000000000000000 P 82d02b2d07714d6791a4888da6376c37 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 82d02b2d07714d6791a4888da6376c37. Latest consensus state: current_term: 1 leader_uuid: "82d02b2d07714d6791a4888da6376c37" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82d02b2d07714d6791a4888da6376c37" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 46229 } } }
I20260620 20:16:16.596185 21427 sys_catalog.cc:458] T 00000000000000000000000000000000 P 82d02b2d07714d6791a4888da6376c37 [sys.catalog]: This master's current role is: LEADER
I20260620 20:16:16.595687 21426 sys_catalog.cc:455] T 00000000000000000000000000000000 P 82d02b2d07714d6791a4888da6376c37 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "82d02b2d07714d6791a4888da6376c37" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82d02b2d07714d6791a4888da6376c37" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 46229 } } }
I20260620 20:16:16.598043 21426 sys_catalog.cc:458] T 00000000000000000000000000000000 P 82d02b2d07714d6791a4888da6376c37 [sys.catalog]: This master's current role is: LEADER
I20260620 20:16:16.602516 21431 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 20:16:16.607980 21431 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 20:16:16.616883 21431 catalog_manager.cc:1347] Generated new cluster ID: a3735ac7d5244a35a1817393c69479b0
I20260620 20:16:16.617003 21431 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 20:16:16.624429 21431 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 20:16:16.625672 21431 catalog_manager.cc:1504] Loading token signing keys...
I20260620 20:16:16.631577 21431 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 82d02b2d07714d6791a4888da6376c37: Generated new TSK 0
I20260620 20:16:16.632334 21431 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.217.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.217.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.217.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.1.keytab.
Entry for principal kudu/127.16.217.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.1.keytab.
I20260620 20:16:16.681715 17252 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
/tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986536687152-17252-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986536687152-17252-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986536687152-17252-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986536687152-17252-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.217.1:0
--local_ip_for_outbound_sockets=127.16.217.1
--webserver_interface=127.16.217.1
--webserver_port=0
--tserver_master_addrs=127.16.217.62:46229
--keytab_file=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.1.keytab
--principal=kudu/127.16.217.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.217.20:33849
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986536687152-17252-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986536687152-17252-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986536687152-17252-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:16:17.008569 21446 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:16:17.008977 21446 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:16:17.009084 21446 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:16:17.016669 21446 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:16:17.016837 21446 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:16:17.017015 21446 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.217.1
I20260620 20:16:17.026985 21446 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.217.20:33849
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986536687152-17252-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986536687152-17252-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.1.keytab
--principal=kudu/127.16.217.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.217.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986536687152-17252-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.217.1
--webserver_port=0
--tserver_master_addrs=127.16.217.62:46229
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.217.1
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986536687152-17252-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:16:17.028187 21446 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:16:17.029465 21446 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:16:17.042102 21454 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:16:17.042168 21452 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:16:17.042168 21451 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:16:17.043073 21446 server_base.cc:1034] running on GCE node
I20260620 20:16:17.043658 21446 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:16:17.044591 21446 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:16:17.045814 21446 hybrid_clock.cc:648] HybridClock initialized: now 1781986577045765 us; error 65 us; skew 500 ppm
I20260620 20:16:17.046130 21446 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:16:17 dist-test-slave-2hpg krb5kdc[21341](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986577, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.217.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:16:17.049113 21446 init.cc:377] Logged in from keytab as kudu/127.16.217.1@KRBTEST.COM (short username kudu)
I20260620 20:16:17.050335 21446 fs_manager.cc:359] Metadata directory not provided
I20260620 20:16:17.050457 21446 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:16:17.050750 21446 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:16:17.053534 21446 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986536687152-17252-0/minicluster-data/ts-0/data/instance:
uuid: "1692ab1ae0874017a5768b9801f5ca90"
format_stamp: "Formatted at 2026-06-20 20:16:17 on dist-test-slave-2hpg"
server_key: "b827d561b35184d1fa6609926fef6683"
server_key_iv: "c9c4ac4079071b22198da0bb2bdca2d6"
server_key_version: "encryptionkey@0"
I20260620 20:16:17.054314 21446 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986536687152-17252-0/minicluster-data/ts-0/wal/instance:
uuid: "1692ab1ae0874017a5768b9801f5ca90"
format_stamp: "Formatted at 2026-06-20 20:16:17 on dist-test-slave-2hpg"
server_key: "b827d561b35184d1fa6609926fef6683"
server_key_iv: "c9c4ac4079071b22198da0bb2bdca2d6"
server_key_version: "encryptionkey@0"
I20260620 20:16:17.059098 21446 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260620 20:16:17.062412 21460 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:16:17.063779 21446 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260620 20:16:17.063972 21446 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986536687152-17252-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986536687152-17252-0/minicluster-data/ts-0/wal
uuid: "1692ab1ae0874017a5768b9801f5ca90"
format_stamp: "Formatted at 2026-06-20 20:16:17 on dist-test-slave-2hpg"
server_key: "b827d561b35184d1fa6609926fef6683"
server_key_iv: "c9c4ac4079071b22198da0bb2bdca2d6"
server_key_version: "encryptionkey@0"
I20260620 20:16:17.064204 21446 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986536687152-17252-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986536687152-17252-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986536687152-17252-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:16:17.082048 21446 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:16:17.086314 21446 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:16:17.086582 21446 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:16:17.087514 21446 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:16:17.088797 21446 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:16:17.088934 21446 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:16:17.089053 21446 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:16:17.089136 21446 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:16:17.122136 21446 rpc_server.cc:307] RPC server started. Bound to: 127.16.217.1:43845
I20260620 20:16:17.122188 21572 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.217.1:43845 every 8 connection(s)
I20260620 20:16:17.122980 21446 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986536687152-17252-0/minicluster-data/ts-0/data/info.pb
I20260620 20:16:17.131004 17252 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu as pid 21446
I20260620 20:16:17.131237 17252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986536687152-17252-0/minicluster-data/ts-0/wal/instance
I20260620 20:16:17.131969 17252 external_mini_cluster.cc:1428] Setting key 920dff4b997baefbd04c23b845c54ca9
Jun 20 20:16:17 dist-test-slave-2hpg krb5kdc[21341](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986577, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.217.1@KRBTEST.COM for kudu/127.16.217.62@KRBTEST.COM
I20260620 20:16:17.142750 21575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:17.125224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.217.1:44603 (local address 127.16.217.62:46229)
0620 20:16:17.125634 (+ 410us) server_negotiation.cc:206] Beginning negotiation
0620 20:16:17.125638 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:16:17.126583 (+ 945us) server_negotiation.cc:407] Connection header received
0620 20:16:17.128423 (+ 1840us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:16:17.128428 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:16:17.128518 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:16:17.128632 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:16:17.130140 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:17.130997 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:17.132671 (+ 1674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:17.133075 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:17.133212 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:17.136594 (+ 3382us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:16:17.136599 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:16:17.136602 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:16:17.136631 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:16:17.139792 (+ 3161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:16:17.140481 (+ 689us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:16:17.140487 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:16:17.140490 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:16:17.140616 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:16:17.141014 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:16:17.141019 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:16:17.141022 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:16:17.141293 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:16:17.141440 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0620 20:16:17.142176 (+ 736us) server_negotiation.cc:299] Negotiation successful
0620 20:16:17.142393 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":317,"thread_start_us":219,"threads_started":1}
I20260620 20:16:17.143689 21576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:17.125665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:46229 (local address 127.16.217.1:44603)
0620 20:16:17.126418 (+ 753us) negotiation.cc:107] Waiting for socket to connect
0620 20:16:17.126458 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 20:16:17.128144 (+ 1686us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:16:17.128854 (+ 710us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:16:17.128863 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:16:17.129379 (+ 516us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:16:17.129983 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:16:17.130001 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:16:17.131144 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:16:17.131151 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:16:17.132532 (+ 1381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:16:17.132543 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:16:17.133320 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:16:17.133327 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:16:17.133611 (+ 284us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:17.134318 (+ 707us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:16:17.134341 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:16:17.136372 (+ 2031us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:16:17.139919 (+ 3547us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:16:17.139927 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:16:17.139944 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:16:17.140364 (+ 420us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:16:17.140717 (+ 353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:16:17.140723 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:16:17.140727 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:16:17.140904 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:16:17.141418 (+ 514us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:16:17.141426 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:16:17.141752 (+ 326us) client_negotiation.cc:769] Sending connection context
0620 20:16:17.142088 (+ 336us) client_negotiation.cc:240] Negotiation successful
0620 20:16:17.142614 (+ 526us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":500,"thread_start_us":323,"threads_started":1}
I20260620 20:16:17.145171 21573 heartbeater.cc:347] Connected to a master server at 127.16.217.62:46229
I20260620 20:16:17.145493 21573 heartbeater.cc:464] Registering TS with master...
I20260620 20:16:17.146330 21573 heartbeater.cc:511] Master 127.16.217.62:46229 requested a full tablet report, sending...
I20260620 20:16:17.148474 21385 ts_manager.cc:194] Registered new tserver with Master: 1692ab1ae0874017a5768b9801f5ca90 (127.16.217.1:43845)
WARNING: no policy specified for kudu/127.16.217.2@KRBTEST.COM; defaulting to no policy
I20260620 20:16:17.150020 21385 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.217.1@KRBTEST.COM'} at 127.16.217.1:44603
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.217.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.217.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.2.keytab.
Entry for principal kudu/127.16.217.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.2.keytab.
I20260620 20:16:17.175350 17252 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
/tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986536687152-17252-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986536687152-17252-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986536687152-17252-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986536687152-17252-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.217.2:0
--local_ip_for_outbound_sockets=127.16.217.2
--webserver_interface=127.16.217.2
--webserver_port=0
--tserver_master_addrs=127.16.217.62:46229
--keytab_file=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.2.keytab
--principal=kudu/127.16.217.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.217.20:33849
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986536687152-17252-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986536687152-17252-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986536687152-17252-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:16:17.503963 21579 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:16:17.504288 21579 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:16:17.504460 21579 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:16:17.511821 21579 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:16:17.511977 21579 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:16:17.512172 21579 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.217.2
I20260620 20:16:17.522356 21579 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.217.20:33849
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986536687152-17252-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986536687152-17252-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.2.keytab
--principal=kudu/127.16.217.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.217.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986536687152-17252-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.217.2
--webserver_port=0
--tserver_master_addrs=127.16.217.62:46229
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.217.2
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986536687152-17252-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:16:17.523705 21579 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:16:17.525077 21579 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:16:17.538609 21584 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:16:17.538604 21585 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:16:17.538794 21587 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:16:17.539580 21579 server_base.cc:1034] running on GCE node
I20260620 20:16:17.540170 21579 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:16:17.541348 21579 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:16:17.542585 21579 hybrid_clock.cc:648] HybridClock initialized: now 1781986577542530 us; error 45 us; skew 500 ppm
I20260620 20:16:17.542976 21579 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:16:17 dist-test-slave-2hpg krb5kdc[21341](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986577, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.217.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:16:17.546171 21579 init.cc:377] Logged in from keytab as kudu/127.16.217.2@KRBTEST.COM (short username kudu)
I20260620 20:16:17.547343 21579 fs_manager.cc:359] Metadata directory not provided
I20260620 20:16:17.547463 21579 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:16:17.547770 21579 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:16:17.551018 21579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986536687152-17252-0/minicluster-data/ts-1/data/instance:
uuid: "3e2fc851b1d64d99b58c10f0cdcaaac7"
format_stamp: "Formatted at 2026-06-20 20:16:17 on dist-test-slave-2hpg"
server_key: "a2fc36c16480f398b996cabe808645bc"
server_key_iv: "f1e7070279b0d1ad0c04731fb465af7e"
server_key_version: "encryptionkey@0"
I20260620 20:16:17.552042 21579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986536687152-17252-0/minicluster-data/ts-1/wal/instance:
uuid: "3e2fc851b1d64d99b58c10f0cdcaaac7"
format_stamp: "Formatted at 2026-06-20 20:16:17 on dist-test-slave-2hpg"
server_key: "a2fc36c16480f398b996cabe808645bc"
server_key_iv: "f1e7070279b0d1ad0c04731fb465af7e"
server_key_version: "encryptionkey@0"
I20260620 20:16:17.558342 21579 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260620 20:16:17.562506 21593 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:16:17.564013 21579 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 20:16:17.564165 21579 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986536687152-17252-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986536687152-17252-0/minicluster-data/ts-1/wal
uuid: "3e2fc851b1d64d99b58c10f0cdcaaac7"
format_stamp: "Formatted at 2026-06-20 20:16:17 on dist-test-slave-2hpg"
server_key: "a2fc36c16480f398b996cabe808645bc"
server_key_iv: "f1e7070279b0d1ad0c04731fb465af7e"
server_key_version: "encryptionkey@0"
I20260620 20:16:17.564395 21579 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986536687152-17252-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986536687152-17252-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986536687152-17252-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:16:17.585206 21579 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:16:17.589133 21579 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:16:17.589388 21579 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:16:17.590359 21579 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:16:17.591626 21579 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:16:17.591735 21579 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:16:17.591866 21579 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:16:17.591928 21579 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:16:17.624538 21579 rpc_server.cc:307] RPC server started. Bound to: 127.16.217.2:39151
I20260620 20:16:17.624578 21705 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.217.2:39151 every 8 connection(s)
I20260620 20:16:17.625427 21579 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986536687152-17252-0/minicluster-data/ts-1/data/info.pb
I20260620 20:16:17.633765 17252 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu as pid 21579
I20260620 20:16:17.633981 17252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986536687152-17252-0/minicluster-data/ts-1/wal/instance
I20260620 20:16:17.634562 17252 external_mini_cluster.cc:1428] Setting key 88d61ceb4eaad9b293bce094aaac6f96
Jun 20 20:16:17 dist-test-slave-2hpg krb5kdc[21341](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986577, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.217.2@KRBTEST.COM for kudu/127.16.217.62@KRBTEST.COM
I20260620 20:16:17.646075 21575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:17.627947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.217.2:55595 (local address 127.16.217.62:46229)
0620 20:16:17.628116 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0620 20:16:17.628121 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:16:17.629254 (+ 1133us) server_negotiation.cc:407] Connection header received
0620 20:16:17.631213 (+ 1959us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:16:17.631219 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:16:17.631315 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:16:17.631432 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:16:17.632962 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:17.633788 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:17.635775 (+ 1987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:17.636426 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:17.636556 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:17.640575 (+ 4019us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:16:17.640579 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:16:17.640583 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:16:17.640611 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:16:17.643066 (+ 2455us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:16:17.643746 (+ 680us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:16:17.643752 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:16:17.643756 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:16:17.643885 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:16:17.644276 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:16:17.644281 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:16:17.644285 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:16:17.644603 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:16:17.644765 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0620 20:16:17.645618 (+ 853us) server_negotiation.cc:299] Negotiation successful
0620 20:16:17.645785 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 20:16:17.647601 21708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:17.628368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:46229 (local address 127.16.217.2:55595)
0620 20:16:17.629082 (+ 714us) negotiation.cc:107] Waiting for socket to connect
0620 20:16:17.629127 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 20:16:17.630926 (+ 1799us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:16:17.631656 (+ 730us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:16:17.631665 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:16:17.632216 (+ 551us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:16:17.632793 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:16:17.632811 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:16:17.633979 (+ 1168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:16:17.633985 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:16:17.635643 (+ 1658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:16:17.635654 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:16:17.636640 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:16:17.636647 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:16:17.636916 (+ 269us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:17.637598 (+ 682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:16:17.637629 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:16:17.640361 (+ 2732us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:16:17.643199 (+ 2838us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:16:17.643208 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:16:17.643223 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:16:17.643632 (+ 409us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:16:17.643989 (+ 357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:16:17.643994 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:16:17.643998 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:16:17.644165 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:16:17.644766 (+ 601us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:16:17.644777 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:16:17.645210 (+ 433us) client_negotiation.cc:769] Sending connection context
0620 20:16:17.645544 (+ 334us) client_negotiation.cc:240] Negotiation successful
0620 20:16:17.646137 (+ 593us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":487,"thread_start_us":295,"threads_started":1}
I20260620 20:16:17.649030 21706 heartbeater.cc:347] Connected to a master server at 127.16.217.62:46229
I20260620 20:16:17.649349 21706 heartbeater.cc:464] Registering TS with master...
I20260620 20:16:17.650071 21706 heartbeater.cc:511] Master 127.16.217.62:46229 requested a full tablet report, sending...
I20260620 20:16:17.651424 21385 ts_manager.cc:194] Registered new tserver with Master: 3e2fc851b1d64d99b58c10f0cdcaaac7 (127.16.217.2:39151)
I20260620 20:16:17.652216 21385 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.217.2@KRBTEST.COM'} at 127.16.217.2:55595
WARNING: no policy specified for kudu/127.16.217.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.217.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.217.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.3.keytab.
Entry for principal kudu/127.16.217.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.3.keytab.
I20260620 20:16:17.678769 17252 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
/tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986536687152-17252-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986536687152-17252-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986536687152-17252-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986536687152-17252-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.217.3:0
--local_ip_for_outbound_sockets=127.16.217.3
--webserver_interface=127.16.217.3
--webserver_port=0
--tserver_master_addrs=127.16.217.62:46229
--keytab_file=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.3.keytab
--principal=kudu/127.16.217.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.217.20:33849
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986536687152-17252-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986536687152-17252-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986536687152-17252-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:16:18.004482 21711 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:16:18.004755 21711 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:16:18.004863 21711 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:16:18.012043 21711 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:16:18.012169 21711 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:16:18.012287 21711 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.217.3
I20260620 20:16:18.022111 21711 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.217.20:33849
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986536687152-17252-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986536687152-17252-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.3.keytab
--principal=kudu/127.16.217.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.217.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986536687152-17252-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.217.3
--webserver_port=0
--tserver_master_addrs=127.16.217.62:46229
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.217.3
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986536687152-17252-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:16:18.023348 21711 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:16:18.024585 21711 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:16:18.037068 21719 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:16:18.037228 21716 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:16:18.037147 21717 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:16:18.038012 21711 server_base.cc:1034] running on GCE node
I20260620 20:16:18.038617 21711 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:16:18.039911 21711 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:16:18.041177 21711 hybrid_clock.cc:648] HybridClock initialized: now 1781986578041115 us; error 39 us; skew 500 ppm
I20260620 20:16:18.041541 21711 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:16:18 dist-test-slave-2hpg krb5kdc[21341](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986578, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.217.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:16:18.052630 21711 init.cc:377] Logged in from keytab as kudu/127.16.217.3@KRBTEST.COM (short username kudu)
I20260620 20:16:18.053841 21711 fs_manager.cc:359] Metadata directory not provided
I20260620 20:16:18.054044 21711 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:16:18.054404 21711 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:16:18.057472 21711 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986536687152-17252-0/minicluster-data/ts-2/data/instance:
uuid: "2fb5cd9e2026426dabf482388c25d2df"
format_stamp: "Formatted at 2026-06-20 20:16:18 on dist-test-slave-2hpg"
server_key: "ec70e52c7a8a79d90ca14af465403cce"
server_key_iv: "7f1758b9388694c0ebf3d648ada0063a"
server_key_version: "encryptionkey@0"
I20260620 20:16:18.058336 21711 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986536687152-17252-0/minicluster-data/ts-2/wal/instance:
uuid: "2fb5cd9e2026426dabf482388c25d2df"
format_stamp: "Formatted at 2026-06-20 20:16:18 on dist-test-slave-2hpg"
server_key: "ec70e52c7a8a79d90ca14af465403cce"
server_key_iv: "7f1758b9388694c0ebf3d648ada0063a"
server_key_version: "encryptionkey@0"
I20260620 20:16:18.063445 21711 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 20:16:18.066903 21725 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:16:18.068279 21711 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 20:16:18.068432 21711 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986536687152-17252-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986536687152-17252-0/minicluster-data/ts-2/wal
uuid: "2fb5cd9e2026426dabf482388c25d2df"
format_stamp: "Formatted at 2026-06-20 20:16:18 on dist-test-slave-2hpg"
server_key: "ec70e52c7a8a79d90ca14af465403cce"
server_key_iv: "7f1758b9388694c0ebf3d648ada0063a"
server_key_version: "encryptionkey@0"
I20260620 20:16:18.068670 21711 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986536687152-17252-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986536687152-17252-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986536687152-17252-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:16:18.097994 21711 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:16:18.102267 21711 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:16:18.102531 21711 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:16:18.103451 21711 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:16:18.104750 21711 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:16:18.104839 21711 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:16:18.104959 21711 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:16:18.105068 21711 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:16:18.137987 21711 rpc_server.cc:307] RPC server started. Bound to: 127.16.217.3:38105
I20260620 20:16:18.138064 21837 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.217.3:38105 every 8 connection(s)
I20260620 20:16:18.138828 21711 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986536687152-17252-0/minicluster-data/ts-2/data/info.pb
I20260620 20:16:18.147835 17252 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu as pid 21711
I20260620 20:16:18.148026 17252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986536687152-17252-0/minicluster-data/ts-2/wal/instance
I20260620 20:16:18.148566 17252 external_mini_cluster.cc:1428] Setting key c65acf0650a053f3268b60de4f6a16e4
Jun 20 20:16:18 dist-test-slave-2hpg krb5kdc[21341](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986578, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.217.3@KRBTEST.COM for kudu/127.16.217.62@KRBTEST.COM
I20260620 20:16:18.152717 21573 heartbeater.cc:503] Master 127.16.217.62:46229 was elected leader, sending a full tablet report...
I20260620 20:16:18.158556 21575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:18.141184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.217.3:59917 (local address 127.16.217.62:46229)
0620 20:16:18.141352 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0620 20:16:18.141356 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:16:18.142454 (+ 1098us) server_negotiation.cc:407] Connection header received
0620 20:16:18.144180 (+ 1726us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:16:18.144185 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:16:18.144278 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:16:18.144380 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:16:18.145830 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:18.146619 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:18.148576 (+ 1957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:18.149035 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:18.149136 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:18.152995 (+ 3859us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:16:18.153001 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:16:18.153006 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:16:18.153046 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:16:18.155715 (+ 2669us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:16:18.156388 (+ 673us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:16:18.156395 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:16:18.156398 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:16:18.156522 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:16:18.156914 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:16:18.156920 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:16:18.156924 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:16:18.157179 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:16:18.157337 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0620 20:16:18.158004 (+ 667us) server_negotiation.cc:299] Negotiation successful
0620 20:16:18.158203 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 20:16:18.159502 21840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:18.141587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:46229 (local address 127.16.217.3:59917)
0620 20:16:18.142302 (+ 715us) negotiation.cc:107] Waiting for socket to connect
0620 20:16:18.142344 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 20:16:18.143920 (+ 1576us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:16:18.144601 (+ 681us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:16:18.144609 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:16:18.145108 (+ 499us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:16:18.145668 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:16:18.145683 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:16:18.146764 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:16:18.146769 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:16:18.148443 (+ 1674us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:16:18.148456 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:16:18.149256 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:16:18.149264 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:16:18.149494 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:18.150338 (+ 844us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:16:18.150365 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:16:18.152724 (+ 2359us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:16:18.155853 (+ 3129us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:16:18.155862 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:16:18.155878 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:16:18.156275 (+ 397us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:16:18.156623 (+ 348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:16:18.156629 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:16:18.156633 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:16:18.156809 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:16:18.157309 (+ 500us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:16:18.157317 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:16:18.157649 (+ 332us) client_negotiation.cc:769] Sending connection context
0620 20:16:18.157932 (+ 283us) client_negotiation.cc:240] Negotiation successful
0620 20:16:18.158425 (+ 493us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":486,"thread_start_us":315,"threads_started":1}
I20260620 20:16:18.160733 21838 heartbeater.cc:347] Connected to a master server at 127.16.217.62:46229
I20260620 20:16:18.161018 21838 heartbeater.cc:464] Registering TS with master...
I20260620 20:16:18.161638 21838 heartbeater.cc:511] Master 127.16.217.62:46229 requested a full tablet report, sending...
I20260620 20:16:18.163223 21385 ts_manager.cc:194] Registered new tserver with Master: 2fb5cd9e2026426dabf482388c25d2df (127.16.217.3:38105)
I20260620 20:16:18.163735 17252 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 20:16:18.164117 21385 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.217.3@KRBTEST.COM'} at 127.16.217.3:59917
I20260620 20:16:18.180156 21575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:18.168341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32838 (local address 127.16.217.62:46229)
0620 20:16:18.168495 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0620 20:16:18.168499 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:16:18.169032 (+ 533us) server_negotiation.cc:407] Connection header received
0620 20:16:18.169241 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:16:18.169244 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:16:18.169303 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:16:18.169395 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:16:18.170774 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:18.171468 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:18.172652 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:18.173053 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:18.173171 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:18.175402 (+ 2231us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:16:18.175407 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:16:18.175410 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:16:18.175437 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:16:18.177730 (+ 2293us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:16:18.178337 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:16:18.178345 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:16:18.178348 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:16:18.178469 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:16:18.178818 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:16:18.178825 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:16:18.178828 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:16:18.179113 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:16:18.179364 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0620 20:16:18.179653 (+ 289us) server_negotiation.cc:299] Negotiation successful
0620 20:16:18.179851 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 20:16:18.183816 21385 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:32838:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 20:16:18.190688 21385 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 20:16:18.212538 21850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:18.202169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.2:39151 (local address 127.0.0.1:51790)
0620 20:16:18.204040 (+ 1871us) negotiation.cc:107] Waiting for socket to connect
0620 20:16:18.204079 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 20:16:18.204350 (+ 271us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:16:18.205450 (+ 1100us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:16:18.205458 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:16:18.205522 (+ 64us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:16:18.206086 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:16:18.206100 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:16:18.207901 (+ 1801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:16:18.207906 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:16:18.209701 (+ 1795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:16:18.209713 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:16:18.211519 (+ 1806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:16:18.211525 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:16:18.211704 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:18.211756 (+ 52us) client_negotiation.cc:769] Sending connection context
0620 20:16:18.211893 (+ 137us) client_negotiation.cc:240] Negotiation successful
0620 20:16:18.212074 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":552,"mutex_wait_us":3,"spinlock_wait_cycles":1408,"thread_start_us":368,"threads_started":1}
I20260620 20:16:18.212846 21853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:18.203193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43402 (local address 127.16.217.3:38105)
0620 20:16:18.204078 (+ 885us) server_negotiation.cc:206] Beginning negotiation
0620 20:16:18.204089 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 20:16:18.204830 (+ 741us) server_negotiation.cc:407] Connection header received
0620 20:16:18.204933 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:16:18.204944 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:16:18.205128 (+ 184us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:16:18.205308 (+ 180us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:16:18.206055 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:18.207791 (+ 1736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:18.210411 (+ 2620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:18.211758 (+ 1347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:18.211893 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:18.212094 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0620 20:16:18.212335 (+ 241us) server_negotiation.cc:299] Negotiation successful
0620 20:16:18.212506 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":688,"thread_start_us":503,"threads_started":1}
I20260620 20:16:18.213524 21851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:18.202397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51790 (local address 127.16.217.2:39151)
0620 20:16:18.203538 (+ 1141us) server_negotiation.cc:206] Beginning negotiation
0620 20:16:18.203548 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 20:16:18.204183 (+ 635us) server_negotiation.cc:407] Connection header received
0620 20:16:18.204630 (+ 447us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:16:18.204638 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:16:18.204907 (+ 269us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:16:18.205087 (+ 180us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:16:18.206270 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:18.207737 (+ 1467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:18.209854 (+ 2117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:18.211083 (+ 1229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:18.211213 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:18.211406 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0620 20:16:18.213181 (+ 1775us) server_negotiation.cc:299] Negotiation successful
0620 20:16:18.213301 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":986,"thread_start_us":993,"threads_started":1}
I20260620 20:16:18.213783 21855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:18.204542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39108 (local address 127.16.217.1:43845)
0620 20:16:18.206033 (+ 1491us) server_negotiation.cc:206] Beginning negotiation
0620 20:16:18.206039 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:16:18.206058 (+ 19us) server_negotiation.cc:407] Connection header received
0620 20:16:18.206124 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:16:18.206128 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:16:18.206312 (+ 184us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:16:18.206500 (+ 188us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:16:18.206973 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:18.208433 (+ 1460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:18.211131 (+ 2698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:18.212438 (+ 1307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:18.212567 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:18.212802 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0620 20:16:18.213295 (+ 493us) server_negotiation.cc:299] Negotiation successful
0620 20:16:18.213459 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1292,"thread_start_us":1106,"threads_started":1}
I20260620 20:16:18.212885 21852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:18.202560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.3:38105 (local address 127.0.0.1:43402)
0620 20:16:18.204522 (+ 1962us) negotiation.cc:107] Waiting for socket to connect
0620 20:16:18.204542 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 20:16:18.204681 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:16:18.205316 (+ 635us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:16:18.205322 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:16:18.205350 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:16:18.205829 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:16:18.205843 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:16:18.207943 (+ 2100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:16:18.207957 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:16:18.210248 (+ 2291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:16:18.210262 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:16:18.211960 (+ 1698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:16:18.211966 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:16:18.212109 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:18.212158 (+ 49us) client_negotiation.cc:769] Sending connection context
0620 20:16:18.212253 (+ 95us) client_negotiation.cc:240] Negotiation successful
0620 20:16:18.212383 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1818,"mutex_wait_us":50,"spinlock_wait_cycles":8576,"thread_start_us":1683,"threads_started":1}
I20260620 20:16:18.213653 21854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:18.203998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.1:43845 (local address 127.0.0.1:39108)
0620 20:16:18.204805 (+ 807us) negotiation.cc:107] Waiting for socket to connect
0620 20:16:18.204821 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 20:16:18.204913 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:16:18.206499 (+ 1586us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:16:18.206504 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:16:18.206530 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:16:18.206826 (+ 296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:16:18.206836 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:16:18.208581 (+ 1745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:16:18.208585 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:16:18.210950 (+ 2365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:16:18.210962 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:16:18.212948 (+ 1986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:16:18.212954 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:16:18.213118 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:18.213147 (+ 29us) client_negotiation.cc:769] Sending connection context
0620 20:16:18.213244 (+ 97us) client_negotiation.cc:240] Negotiation successful
0620 20:16:18.213357 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":665,"thread_start_us":904,"threads_started":1}
I20260620 20:16:18.217526 21508 tablet_service.cc:1467] Processing CreateTablet for tablet 33d495720f10491895e28fee78651d4c (DEFAULT_TABLE table=test-table [id=ae58730ffb7942be9cf3d6bb9d479061]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:16:18.217531 21773 tablet_service.cc:1467] Processing CreateTablet for tablet 33d495720f10491895e28fee78651d4c (DEFAULT_TABLE table=test-table [id=ae58730ffb7942be9cf3d6bb9d479061]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:16:18.217531 21641 tablet_service.cc:1467] Processing CreateTablet for tablet 33d495720f10491895e28fee78651d4c (DEFAULT_TABLE table=test-table [id=ae58730ffb7942be9cf3d6bb9d479061]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:16:18.218812 21508 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 33d495720f10491895e28fee78651d4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:16:18.218858 21641 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 33d495720f10491895e28fee78651d4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:16:18.218849 21773 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 33d495720f10491895e28fee78651d4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:16:18.229569 21856 tablet_bootstrap.cc:492] T 33d495720f10491895e28fee78651d4c P 1692ab1ae0874017a5768b9801f5ca90: Bootstrap starting.
I20260620 20:16:18.233526 21856 tablet_bootstrap.cc:654] T 33d495720f10491895e28fee78651d4c P 1692ab1ae0874017a5768b9801f5ca90: Neither blocks nor log segments found. Creating new log.
I20260620 20:16:18.233786 21857 tablet_bootstrap.cc:492] T 33d495720f10491895e28fee78651d4c P 3e2fc851b1d64d99b58c10f0cdcaaac7: Bootstrap starting.
I20260620 20:16:18.234549 21858 tablet_bootstrap.cc:492] T 33d495720f10491895e28fee78651d4c P 2fb5cd9e2026426dabf482388c25d2df: Bootstrap starting.
I20260620 20:16:18.234910 21856 log.cc:824] T 33d495720f10491895e28fee78651d4c P 1692ab1ae0874017a5768b9801f5ca90: Log is configured to *not* fsync() on all Append() calls
I20260620 20:16:18.237322 21857 tablet_bootstrap.cc:654] T 33d495720f10491895e28fee78651d4c P 3e2fc851b1d64d99b58c10f0cdcaaac7: Neither blocks nor log segments found. Creating new log.
I20260620 20:16:18.237605 21858 tablet_bootstrap.cc:654] T 33d495720f10491895e28fee78651d4c P 2fb5cd9e2026426dabf482388c25d2df: Neither blocks nor log segments found. Creating new log.
I20260620 20:16:18.238320 21856 tablet_bootstrap.cc:492] T 33d495720f10491895e28fee78651d4c P 1692ab1ae0874017a5768b9801f5ca90: No bootstrap required, opened a new log
I20260620 20:16:18.238583 21856 ts_tablet_manager.cc:1397] T 33d495720f10491895e28fee78651d4c P 1692ab1ae0874017a5768b9801f5ca90: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260620 20:16:18.238610 21857 log.cc:824] T 33d495720f10491895e28fee78651d4c P 3e2fc851b1d64d99b58c10f0cdcaaac7: Log is configured to *not* fsync() on all Append() calls
I20260620 20:16:18.238785 21858 log.cc:824] T 33d495720f10491895e28fee78651d4c P 2fb5cd9e2026426dabf482388c25d2df: Log is configured to *not* fsync() on all Append() calls
I20260620 20:16:18.241716 21857 tablet_bootstrap.cc:492] T 33d495720f10491895e28fee78651d4c P 3e2fc851b1d64d99b58c10f0cdcaaac7: No bootstrap required, opened a new log
I20260620 20:16:18.242089 21857 ts_tablet_manager.cc:1397] T 33d495720f10491895e28fee78651d4c P 3e2fc851b1d64d99b58c10f0cdcaaac7: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260620 20:16:18.242319 21858 tablet_bootstrap.cc:492] T 33d495720f10491895e28fee78651d4c P 2fb5cd9e2026426dabf482388c25d2df: No bootstrap required, opened a new log
I20260620 20:16:18.242555 21858 ts_tablet_manager.cc:1397] T 33d495720f10491895e28fee78651d4c P 2fb5cd9e2026426dabf482388c25d2df: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.006s
I20260620 20:16:18.249112 21858 raft_consensus.cc:348] T 33d495720f10491895e28fee78651d4c P 2fb5cd9e2026426dabf482388c25d2df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e2fc851b1d64d99b58c10f0cdcaaac7" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 39151 } } peers { permanent_uuid: "2fb5cd9e2026426dabf482388c25d2df" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 38105 } } peers { permanent_uuid: "1692ab1ae0874017a5768b9801f5ca90" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 43845 } }
I20260620 20:16:18.248991 21856 raft_consensus.cc:348] T 33d495720f10491895e28fee78651d4c P 1692ab1ae0874017a5768b9801f5ca90 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e2fc851b1d64d99b58c10f0cdcaaac7" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 39151 } } peers { permanent_uuid: "2fb5cd9e2026426dabf482388c25d2df" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 38105 } } peers { permanent_uuid: "1692ab1ae0874017a5768b9801f5ca90" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 43845 } }
I20260620 20:16:18.249444 21858 raft_consensus.cc:374] T 33d495720f10491895e28fee78651d4c P 2fb5cd9e2026426dabf482388c25d2df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:16:18.249441 21856 raft_consensus.cc:374] T 33d495720f10491895e28fee78651d4c P 1692ab1ae0874017a5768b9801f5ca90 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:16:18.249532 21856 raft_consensus.cc:729] T 33d495720f10491895e28fee78651d4c P 1692ab1ae0874017a5768b9801f5ca90 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1692ab1ae0874017a5768b9801f5ca90, State: Initialized, Role: FOLLOWER
I20260620 20:16:18.249572 21858 raft_consensus.cc:729] T 33d495720f10491895e28fee78651d4c P 2fb5cd9e2026426dabf482388c25d2df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2fb5cd9e2026426dabf482388c25d2df, State: Initialized, Role: FOLLOWER
I20260620 20:16:18.250218 21858 consensus_queue.cc:260] T 33d495720f10491895e28fee78651d4c P 2fb5cd9e2026426dabf482388c25d2df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e2fc851b1d64d99b58c10f0cdcaaac7" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 39151 } } peers { permanent_uuid: "2fb5cd9e2026426dabf482388c25d2df" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 38105 } } peers { permanent_uuid: "1692ab1ae0874017a5768b9801f5ca90" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 43845 } }
I20260620 20:16:18.250162 21856 consensus_queue.cc:260] T 33d495720f10491895e28fee78651d4c P 1692ab1ae0874017a5768b9801f5ca90 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e2fc851b1d64d99b58c10f0cdcaaac7" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 39151 } } peers { permanent_uuid: "2fb5cd9e2026426dabf482388c25d2df" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 38105 } } peers { permanent_uuid: "1692ab1ae0874017a5768b9801f5ca90" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 43845 } }
I20260620 20:16:18.251775 21838 heartbeater.cc:503] Master 127.16.217.62:46229 was elected leader, sending a full tablet report...
I20260620 20:16:18.252014 21858 ts_tablet_manager.cc:1428] T 33d495720f10491895e28fee78651d4c P 2fb5cd9e2026426dabf482388c25d2df: Time spent starting tablet: real 0.009s user 0.004s sys 0.004s
I20260620 20:16:18.252259 21856 ts_tablet_manager.cc:1428] T 33d495720f10491895e28fee78651d4c P 1692ab1ae0874017a5768b9801f5ca90: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
I20260620 20:16:18.252445 21857 raft_consensus.cc:348] T 33d495720f10491895e28fee78651d4c P 3e2fc851b1d64d99b58c10f0cdcaaac7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e2fc851b1d64d99b58c10f0cdcaaac7" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 39151 } } peers { permanent_uuid: "2fb5cd9e2026426dabf482388c25d2df" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 38105 } } peers { permanent_uuid: "1692ab1ae0874017a5768b9801f5ca90" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 43845 } }
I20260620 20:16:18.252934 21857 raft_consensus.cc:374] T 33d495720f10491895e28fee78651d4c P 3e2fc851b1d64d99b58c10f0cdcaaac7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:16:18.253072 21857 raft_consensus.cc:729] T 33d495720f10491895e28fee78651d4c P 3e2fc851b1d64d99b58c10f0cdcaaac7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e2fc851b1d64d99b58c10f0cdcaaac7, State: Initialized, Role: FOLLOWER
I20260620 20:16:18.254742 21857 consensus_queue.cc:260] T 33d495720f10491895e28fee78651d4c P 3e2fc851b1d64d99b58c10f0cdcaaac7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e2fc851b1d64d99b58c10f0cdcaaac7" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 39151 } } peers { permanent_uuid: "2fb5cd9e2026426dabf482388c25d2df" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 38105 } } peers { permanent_uuid: "1692ab1ae0874017a5768b9801f5ca90" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 43845 } }
I20260620 20:16:18.255749 21706 heartbeater.cc:503] Master 127.16.217.62:46229 was elected leader, sending a full tablet report...
I20260620 20:16:18.256381 21857 ts_tablet_manager.cc:1428] T 33d495720f10491895e28fee78651d4c P 3e2fc851b1d64d99b58c10f0cdcaaac7: Time spent starting tablet: real 0.014s user 0.012s sys 0.000s
I20260620 20:16:18.341753 21862 raft_consensus.cc:482] T 33d495720f10491895e28fee78651d4c P 2fb5cd9e2026426dabf482388c25d2df [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 20:16:18.342110 21862 raft_consensus.cc:504] T 33d495720f10491895e28fee78651d4c P 2fb5cd9e2026426dabf482388c25d2df [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e2fc851b1d64d99b58c10f0cdcaaac7" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 39151 } } peers { permanent_uuid: "2fb5cd9e2026426dabf482388c25d2df" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 38105 } } peers { permanent_uuid: "1692ab1ae0874017a5768b9801f5ca90" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 43845 } }
I20260620 20:16:18.343833 21862 leader_election.cc:288] T 33d495720f10491895e28fee78651d4c P 2fb5cd9e2026426dabf482388c25d2df [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3e2fc851b1d64d99b58c10f0cdcaaac7 (127.16.217.2:39151), 1692ab1ae0874017a5768b9801f5ca90 (127.16.217.1:43845)
I20260620 20:16:18.349334 21851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:18.344192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.217.3:53487 (local address 127.16.217.2:39151)
0620 20:16:18.344407 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0620 20:16:18.344414 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:16:18.344714 (+ 300us) server_negotiation.cc:407] Connection header received
0620 20:16:18.344869 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:16:18.344880 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:16:18.344936 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:16:18.345030 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:16:18.345410 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:18.346141 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:18.347718 (+ 1577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:18.348449 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:18.348609 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:18.348689 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0620 20:16:18.348933 (+ 244us) server_negotiation.cc:299] Negotiation successful
0620 20:16:18.349047 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260620 20:16:18.349350 21840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:18.344383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.2:39151 (local address 127.16.217.3:53487)
0620 20:16:18.344587 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0620 20:16:18.344610 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 20:16:18.344727 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:16:18.345051 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:16:18.345055 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:16:18.345087 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:16:18.345291 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:16:18.345304 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:16:18.346285 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:16:18.346290 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:16:18.347574 (+ 1284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:16:18.347586 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:16:18.348592 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:16:18.348598 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:16:18.348789 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:18.348812 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 20:16:18.348911 (+ 99us) client_negotiation.cc:240] Negotiation successful
0620 20:16:18.349038 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":68}
I20260620 20:16:18.350106 21855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:18.344258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.217.3:45933 (local address 127.16.217.1:43845)
0620 20:16:18.344465 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0620 20:16:18.344473 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 20:16:18.345313 (+ 840us) server_negotiation.cc:407] Connection header received
0620 20:16:18.345476 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:16:18.345480 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:16:18.345535 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:16:18.345627 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:16:18.346150 (+ 523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:18.346866 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:18.348474 (+ 1608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:18.349176 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:18.349359 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:18.349428 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 20:16:18.349761 (+ 333us) server_negotiation.cc:299] Negotiation successful
0620 20:16:18.349907 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260620 20:16:18.350188 21865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:18.344414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.1:43845 (local address 127.16.217.3:45933)
0620 20:16:18.345215 (+ 801us) negotiation.cc:107] Waiting for socket to connect
0620 20:16:18.345231 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 20:16:18.345331 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:16:18.345641 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:16:18.345645 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:16:18.345665 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:16:18.346014 (+ 349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:16:18.346022 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:16:18.347003 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:16:18.347008 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:16:18.348329 (+ 1321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:16:18.348339 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:16:18.349365 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:16:18.349371 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:16:18.349591 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:18.349619 (+ 28us) client_negotiation.cc:769] Sending connection context
0620 20:16:18.349704 (+ 85us) client_negotiation.cc:240] Negotiation successful
0620 20:16:18.349834 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":697,"thread_start_us":489,"threads_started":1}
I20260620 20:16:18.350600 21661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33d495720f10491895e28fee78651d4c" candidate_uuid: "2fb5cd9e2026426dabf482388c25d2df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e2fc851b1d64d99b58c10f0cdcaaac7" is_pre_election: true
I20260620 20:16:18.351140 21661 raft_consensus.cc:2454] T 33d495720f10491895e28fee78651d4c P 3e2fc851b1d64d99b58c10f0cdcaaac7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2fb5cd9e2026426dabf482388c25d2df in term 0.
I20260620 20:16:18.351145 21528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33d495720f10491895e28fee78651d4c" candidate_uuid: "2fb5cd9e2026426dabf482388c25d2df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1692ab1ae0874017a5768b9801f5ca90" is_pre_election: true
I20260620 20:16:18.351673 21528 raft_consensus.cc:2454] T 33d495720f10491895e28fee78651d4c P 1692ab1ae0874017a5768b9801f5ca90 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2fb5cd9e2026426dabf482388c25d2df in term 0.
I20260620 20:16:18.351773 21727 leader_election.cc:302] T 33d495720f10491895e28fee78651d4c P 2fb5cd9e2026426dabf482388c25d2df [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2fb5cd9e2026426dabf482388c25d2df, 3e2fc851b1d64d99b58c10f0cdcaaac7; no voters:
I20260620 20:16:18.352206 21862 raft_consensus.cc:2781] T 33d495720f10491895e28fee78651d4c P 2fb5cd9e2026426dabf482388c25d2df [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 20:16:18.352360 21862 raft_consensus.cc:482] T 33d495720f10491895e28fee78651d4c P 2fb5cd9e2026426dabf482388c25d2df [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 20:16:18.352471 21862 raft_consensus.cc:3037] T 33d495720f10491895e28fee78651d4c P 2fb5cd9e2026426dabf482388c25d2df [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:16:18.354607 21862 raft_consensus.cc:504] T 33d495720f10491895e28fee78651d4c P 2fb5cd9e2026426dabf482388c25d2df [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e2fc851b1d64d99b58c10f0cdcaaac7" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 39151 } } peers { permanent_uuid: "2fb5cd9e2026426dabf482388c25d2df" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 38105 } } peers { permanent_uuid: "1692ab1ae0874017a5768b9801f5ca90" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 43845 } }
I20260620 20:16:18.355198 21862 leader_election.cc:288] T 33d495720f10491895e28fee78651d4c P 2fb5cd9e2026426dabf482388c25d2df [CANDIDATE]: Term 1 election: Requested vote from peers 3e2fc851b1d64d99b58c10f0cdcaaac7 (127.16.217.2:39151), 1692ab1ae0874017a5768b9801f5ca90 (127.16.217.1:43845)
I20260620 20:16:18.355568 21661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33d495720f10491895e28fee78651d4c" candidate_uuid: "2fb5cd9e2026426dabf482388c25d2df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e2fc851b1d64d99b58c10f0cdcaaac7"
I20260620 20:16:18.355638 21528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33d495720f10491895e28fee78651d4c" candidate_uuid: "2fb5cd9e2026426dabf482388c25d2df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1692ab1ae0874017a5768b9801f5ca90"
I20260620 20:16:18.355775 21661 raft_consensus.cc:3037] T 33d495720f10491895e28fee78651d4c P 3e2fc851b1d64d99b58c10f0cdcaaac7 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:16:18.355825 21528 raft_consensus.cc:3037] T 33d495720f10491895e28fee78651d4c P 1692ab1ae0874017a5768b9801f5ca90 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:16:18.359010 21661 raft_consensus.cc:2454] T 33d495720f10491895e28fee78651d4c P 3e2fc851b1d64d99b58c10f0cdcaaac7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2fb5cd9e2026426dabf482388c25d2df in term 1.
I20260620 20:16:18.359010 21528 raft_consensus.cc:2454] T 33d495720f10491895e28fee78651d4c P 1692ab1ae0874017a5768b9801f5ca90 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2fb5cd9e2026426dabf482388c25d2df in term 1.
I20260620 20:16:18.359378 21727 leader_election.cc:302] T 33d495720f10491895e28fee78651d4c P 2fb5cd9e2026426dabf482388c25d2df [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2fb5cd9e2026426dabf482388c25d2df, 3e2fc851b1d64d99b58c10f0cdcaaac7; no voters:
I20260620 20:16:18.359676 21862 raft_consensus.cc:2781] T 33d495720f10491895e28fee78651d4c P 2fb5cd9e2026426dabf482388c25d2df [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:16:18.360080 21862 raft_consensus.cc:686] T 33d495720f10491895e28fee78651d4c P 2fb5cd9e2026426dabf482388c25d2df [term 1 LEADER]: Becoming Leader. State: Replica: 2fb5cd9e2026426dabf482388c25d2df, State: Running, Role: LEADER
I20260620 20:16:18.360533 21862 consensus_queue.cc:237] T 33d495720f10491895e28fee78651d4c P 2fb5cd9e2026426dabf482388c25d2df [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e2fc851b1d64d99b58c10f0cdcaaac7" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 39151 } } peers { permanent_uuid: "2fb5cd9e2026426dabf482388c25d2df" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 38105 } } peers { permanent_uuid: "1692ab1ae0874017a5768b9801f5ca90" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 43845 } }
I20260620 20:16:18.364954 21384 catalog_manager.cc:5495] T 33d495720f10491895e28fee78651d4c P 2fb5cd9e2026426dabf482388c25d2df reported cstate change: term changed from 0 to 1, leader changed from <none> to 2fb5cd9e2026426dabf482388c25d2df (127.16.217.3). New cstate: current_term: 1 leader_uuid: "2fb5cd9e2026426dabf482388c25d2df" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e2fc851b1d64d99b58c10f0cdcaaac7" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 39151 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2fb5cd9e2026426dabf482388c25d2df" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 38105 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1692ab1ae0874017a5768b9801f5ca90" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 43845 } health_report { overall_health: UNKNOWN } } }
W20260620 20:16:18.375166 21574 tablet.cc:2369] T 33d495720f10491895e28fee78651d4c P 1692ab1ae0874017a5768b9801f5ca90: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 20:16:18.377241 21707 tablet.cc:2369] T 33d495720f10491895e28fee78651d4c P 3e2fc851b1d64d99b58c10f0cdcaaac7: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 20:16:18.390208 21839 tablet.cc:2369] T 33d495720f10491895e28fee78651d4c P 2fb5cd9e2026426dabf482388c25d2df: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 20:16:18.405377 21853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:18.400223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43414 (local address 127.16.217.3:38105)
0620 20:16:18.400515 (+ 292us) server_negotiation.cc:206] Beginning negotiation
0620 20:16:18.400521 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:16:18.400538 (+ 17us) server_negotiation.cc:407] Connection header received
0620 20:16:18.400725 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:16:18.400729 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:16:18.400799 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:16:18.400941 (+ 142us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 20:16:18.401266 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:18.402096 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:18.403437 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:18.403821 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:18.403941 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:18.404374 (+ 433us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 20:16:18.404832 (+ 458us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 20:16:18.405009 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0620 20:16:18.405058 (+ 49us) server_negotiation.cc:299] Negotiation successful
0620 20:16:18.405129 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152}
I20260620 20:16:18.413404 21661 raft_consensus.cc:1261] T 33d495720f10491895e28fee78651d4c P 3e2fc851b1d64d99b58c10f0cdcaaac7 [term 1 FOLLOWER]: Refusing update from remote peer 2fb5cd9e2026426dabf482388c25d2df: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:16:18.413398 21528 raft_consensus.cc:1261] T 33d495720f10491895e28fee78651d4c P 1692ab1ae0874017a5768b9801f5ca90 [term 1 FOLLOWER]: Refusing update from remote peer 2fb5cd9e2026426dabf482388c25d2df: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:16:18.414467 21862 consensus_queue.cc:1035] T 33d495720f10491895e28fee78651d4c P 2fb5cd9e2026426dabf482388c25d2df [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e2fc851b1d64d99b58c10f0cdcaaac7" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 39151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:16:18.414791 21866 consensus_queue.cc:1035] T 33d495720f10491895e28fee78651d4c P 2fb5cd9e2026426dabf482388c25d2df [LEADER]: Connected to new peer: Peer: permanent_uuid: "1692ab1ae0874017a5768b9801f5ca90" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 43845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:16:18.425217 21869 mvcc.cc:205] Tried to move back new op lower bound from 7299017025170202624 to 7299017024968122368. Current Snapshot: MvccSnapshot[applied={T|T < 7299017025170202624}]
I20260620 20:16:18.425678 21871 mvcc.cc:205] Tried to move back new op lower bound from 7299017025170202624 to 7299017024968122368. Current Snapshot: MvccSnapshot[applied={T|T < 7299017025170202624}]
I20260620 20:16:18.432567 21870 mvcc.cc:205] Tried to move back new op lower bound from 7299017025170202624 to 7299017024968122368. Current Snapshot: MvccSnapshot[applied={T|T < 7299017025170202624}]
I20260620 20:16:18.440464 21384 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:32838:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 20:16:18.440734 21384 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:32838:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 20:16:18.444499 21384 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 82d02b2d07714d6791a4888da6376c37: Sending DeleteTablet for 3 replicas of tablet 33d495720f10491895e28fee78651d4c
I20260620 20:16:18.445665 21773 tablet_service.cc:1514] Processing DeleteTablet for tablet 33d495720f10491895e28fee78651d4c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:16:18 UTC) from {username='kudu', principal='kudu/127.16.217.62@KRBTEST.COM'} at 127.0.0.1:43402
I20260620 20:16:18.445768 21641 tablet_service.cc:1514] Processing DeleteTablet for tablet 33d495720f10491895e28fee78651d4c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:16:18 UTC) from {username='kudu', principal='kudu/127.16.217.62@KRBTEST.COM'} at 127.0.0.1:51790
I20260620 20:16:18.446213 21508 tablet_service.cc:1514] Processing DeleteTablet for tablet 33d495720f10491895e28fee78651d4c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:16:18 UTC) from {username='kudu', principal='kudu/127.16.217.62@KRBTEST.COM'} at 127.0.0.1:39108
I20260620 20:16:18.448947 17252 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu pid 21446
I20260620 20:16:18.461162 21881 tablet_replica.cc:330] T 33d495720f10491895e28fee78651d4c P 3e2fc851b1d64d99b58c10f0cdcaaac7: stopping tablet replica
I20260620 20:16:18.462023 21881 raft_consensus.cc:2229] T 33d495720f10491895e28fee78651d4c P 3e2fc851b1d64d99b58c10f0cdcaaac7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 20:16:18.462812 21881 raft_consensus.cc:2258] T 33d495720f10491895e28fee78651d4c P 3e2fc851b1d64d99b58c10f0cdcaaac7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:16:18.464895 21882 tablet_replica.cc:330] T 33d495720f10491895e28fee78651d4c P 1692ab1ae0874017a5768b9801f5ca90: stopping tablet replica
I20260620 20:16:18.465448 21882 raft_consensus.cc:2229] T 33d495720f10491895e28fee78651d4c P 1692ab1ae0874017a5768b9801f5ca90 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 20:16:18.466107 21882 raft_consensus.cc:2258] T 33d495720f10491895e28fee78651d4c P 1692ab1ae0874017a5768b9801f5ca90 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:16:18.467147 21880 tablet_replica.cc:330] T 33d495720f10491895e28fee78651d4c P 2fb5cd9e2026426dabf482388c25d2df: stopping tablet replica
I20260620 20:16:18.468358 21881 ts_tablet_manager.cc:1905] T 33d495720f10491895e28fee78651d4c P 3e2fc851b1d64d99b58c10f0cdcaaac7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:16:18.469488 21880 raft_consensus.cc:2229] T 33d495720f10491895e28fee78651d4c P 2fb5cd9e2026426dabf482388c25d2df [term 1 LEADER]: Raft consensus shutting down.
I20260620 20:16:18.470180 21882 ts_tablet_manager.cc:1905] T 33d495720f10491895e28fee78651d4c P 1692ab1ae0874017a5768b9801f5ca90: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:16:18.471848 21880 raft_consensus.cc:2258] T 33d495720f10491895e28fee78651d4c P 2fb5cd9e2026426dabf482388c25d2df [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:16:18.475728 21881 ts_tablet_manager.cc:1918] T 33d495720f10491895e28fee78651d4c P 3e2fc851b1d64d99b58c10f0cdcaaac7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 20:16:18.475903 21881 log.cc:1196] T 33d495720f10491895e28fee78651d4c P 3e2fc851b1d64d99b58c10f0cdcaaac7: Deleting WAL directory at /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986536687152-17252-0/minicluster-data/ts-1/wal/wals/33d495720f10491895e28fee78651d4c
I20260620 20:16:18.476428 21881 ts_tablet_manager.cc:1939] T 33d495720f10491895e28fee78651d4c P 3e2fc851b1d64d99b58c10f0cdcaaac7: Deleting consensus metadata
Jun 20 20:16:18 dist-test-slave-2hpg krb5kdc[21341](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986576, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.217.1@KRBTEST.COM
I20260620 20:16:18.478399 21370 catalog_manager.cc:4841] TS 3e2fc851b1d64d99b58c10f0cdcaaac7 (127.16.217.2:39151): tablet 33d495720f10491895e28fee78651d4c (table test-table [id=ae58730ffb7942be9cf3d6bb9d479061]) successfully deleted
I20260620 20:16:18.479346 21880 ts_tablet_manager.cc:1905] T 33d495720f10491895e28fee78651d4c P 2fb5cd9e2026426dabf482388c25d2df: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:16:18.482493 21882 ts_tablet_manager.cc:1918] T 33d495720f10491895e28fee78651d4c P 1692ab1ae0874017a5768b9801f5ca90: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 20:16:18.482746 21882 log.cc:1196] T 33d495720f10491895e28fee78651d4c P 1692ab1ae0874017a5768b9801f5ca90: Deleting WAL directory at /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986536687152-17252-0/minicluster-data/ts-0/wal/wals/33d495720f10491895e28fee78651d4c
I20260620 20:16:18.483310 21882 ts_tablet_manager.cc:1939] T 33d495720f10491895e28fee78651d4c P 1692ab1ae0874017a5768b9801f5ca90: Deleting consensus metadata
I20260620 20:16:18.485132 21369 catalog_manager.cc:4841] TS 1692ab1ae0874017a5768b9801f5ca90 (127.16.217.1:43845): tablet 33d495720f10491895e28fee78651d4c (table test-table [id=ae58730ffb7942be9cf3d6bb9d479061]) successfully deleted
I20260620 20:16:18.487150 21855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:18.451148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39120 (local address 127.16.217.1:43845)
0620 20:16:18.451411 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0620 20:16:18.451423 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 20:16:18.464123 (+ 12700us) server_negotiation.cc:407] Connection header received
0620 20:16:18.464380 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:16:18.464390 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:16:18.464537 (+ 147us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:16:18.464685 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:16:18.467021 (+ 2336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:18.468410 (+ 1389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:18.470595 (+ 2185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:18.471355 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:18.471476 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:18.478032 (+ 6556us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:16:18.478052 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:16:18.478075 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:16:18.478126 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:16:18.482889 (+ 4763us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:16:18.483968 (+ 1079us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:16:18.483979 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:16:18.483983 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:16:18.484196 (+ 213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:16:18.484797 (+ 601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:16:18.484804 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:16:18.484808 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:16:18.485477 (+ 669us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:16:18.485897 (+ 420us) server_negotiation.cc:1035] Waiting for connection context
0620 20:16:18.486203 (+ 306us) server_negotiation.cc:299] Negotiation successful
0620 20:16:18.486456 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260620 20:16:18.487841 21568 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:16:18.488027 21880 ts_tablet_manager.cc:1918] T 33d495720f10491895e28fee78651d4c P 2fb5cd9e2026426dabf482388c25d2df: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 20:16:18.488265 21880 log.cc:1196] T 33d495720f10491895e28fee78651d4c P 2fb5cd9e2026426dabf482388c25d2df: Deleting WAL directory at /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986536687152-17252-0/minicluster-data/ts-2/wal/wals/33d495720f10491895e28fee78651d4c
I20260620 20:16:18.488952 21880 ts_tablet_manager.cc:1939] T 33d495720f10491895e28fee78651d4c P 2fb5cd9e2026426dabf482388c25d2df: Deleting consensus metadata
I20260620 20:16:18.491353 21371 catalog_manager.cc:4841] TS 2fb5cd9e2026426dabf482388c25d2df (127.16.217.3:38105): tablet 33d495720f10491895e28fee78651d4c (table test-table [id=ae58730ffb7942be9cf3d6bb9d479061]) successfully deleted
I20260620 20:16:18.583998 17252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu with pid 21446
I20260620 20:16:18.603804 17252 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu pid 21579
Jun 20 20:16:18 dist-test-slave-2hpg krb5kdc[21341](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986576, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.217.2@KRBTEST.COM
I20260620 20:16:18.619014 21851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:18.604601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51804 (local address 127.16.217.2:39151)
0620 20:16:18.604830 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0620 20:16:18.604836 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:16:18.604856 (+ 20us) server_negotiation.cc:407] Connection header received
0620 20:16:18.605010 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:16:18.605015 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:16:18.605084 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:16:18.605184 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:16:18.606471 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:18.607197 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:18.608410 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:18.608792 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:18.608911 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:18.613245 (+ 4334us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:16:18.613259 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:16:18.613297 (+ 38us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:16:18.613331 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:16:18.616179 (+ 2848us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:16:18.616959 (+ 780us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:16:18.616970 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:16:18.616976 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:16:18.617141 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:16:18.617526 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:16:18.617530 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:16:18.617532 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:16:18.618006 (+ 474us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:16:18.618333 (+ 327us) server_negotiation.cc:1035] Waiting for connection context
0620 20:16:18.618554 (+ 221us) server_negotiation.cc:299] Negotiation successful
0620 20:16:18.618716 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
I20260620 20:16:18.619683 21701 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:16:18.709230 17252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu with pid 21579
I20260620 20:16:18.728276 17252 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu pid 21711
Jun 20 20:16:18 dist-test-slave-2hpg krb5kdc[21341](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986576, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.217.3@KRBTEST.COM
I20260620 20:16:18.747256 21853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:18.729168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43420 (local address 127.16.217.3:38105)
0620 20:16:18.729369 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0620 20:16:18.729374 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:16:18.729463 (+ 89us) server_negotiation.cc:407] Connection header received
0620 20:16:18.729658 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:16:18.729661 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:16:18.729741 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:16:18.729826 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:16:18.731607 (+ 1781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:18.732519 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:18.734448 (+ 1929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:18.735037 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:18.735154 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:18.740688 (+ 5534us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:16:18.740703 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:16:18.740719 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:16:18.740764 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:16:18.744140 (+ 3376us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:16:18.744919 (+ 779us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:16:18.744927 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:16:18.744930 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:16:18.745099 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:16:18.745520 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:16:18.745530 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:16:18.745533 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:16:18.745980 (+ 447us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:16:18.746342 (+ 362us) server_negotiation.cc:1035] Waiting for connection context
0620 20:16:18.746624 (+ 282us) server_negotiation.cc:299] Negotiation successful
0620 20:16:18.746872 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 20:16:18.747982 21833 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:16:18.869916 17252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu with pid 21711
I20260620 20:16:18.889169 17252 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu pid 21354
I20260620 20:16:18.890559 21415 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:16:19.004285 17252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu with pid 21354
2026-06-20T20:16:19Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (4151 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-06-20T20:16:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T20:16:19Z Disabled control of system clock
I20260620 20:16:19.071317 17252 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
/tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986536687152-17252-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986536687152-17252-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986536687152-17252-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986536687152-17252-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.217.62:40285
--webserver_interface=127.16.217.62
--webserver_port=0
--builtin_ntp_servers=127.16.217.20:46727
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.217.62:40285
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260620 20:16:19.392632 21891 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:16:19.392920 21891 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:16:19.392998 21891 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:16:19.400177 21891 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:16:19.400285 21891 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:16:19.400331 21891 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:16:19.400378 21891 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:16:19.400422 21891 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:16:19.409586 21891 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.217.20:46727
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986536687152-17252-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986536687152-17252-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.217.62:40285
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.217.62:40285
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986536687152-17252-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.217.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986536687152-17252-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:16:19.410921 21891 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:16:19.412658 21891 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:16:19.424599 21899 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:16:19.424705 21896 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:16:19.424656 21897 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:16:19.425180 21891 server_base.cc:1034] running on GCE node
I20260620 20:16:19.426149 21891 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:16:19.427672 21891 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:16:19.428931 21891 hybrid_clock.cc:648] HybridClock initialized: now 1781986579428856 us; error 80 us; skew 500 ppm
I20260620 20:16:19.429361 21891 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 20:16:19.432086 21891 webserver.cc:460] Webserver started at http://127.16.217.62:33365/ using document root <none> and password file <none>
I20260620 20:16:19.432871 21891 fs_manager.cc:359] Metadata directory not provided
I20260620 20:16:19.433003 21891 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:16:19.433405 21891 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:16:19.436668 21891 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986536687152-17252-0/minicluster-data/master-0/data/instance:
uuid: "dbea80af3cbf4ca3915c6736ad52aafb"
format_stamp: "Formatted at 2026-06-20 20:16:19 on dist-test-slave-2hpg"
server_key: "7f6bffc79569d8e001936c5fd9d8c143"
server_key_iv: "b5fa0942500f3afc0100a8556ca36a99"
server_key_version: "encryptionkey@0"
I20260620 20:16:19.437602 21891 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986536687152-17252-0/minicluster-data/master-0/wal/instance:
uuid: "dbea80af3cbf4ca3915c6736ad52aafb"
format_stamp: "Formatted at 2026-06-20 20:16:19 on dist-test-slave-2hpg"
server_key: "7f6bffc79569d8e001936c5fd9d8c143"
server_key_iv: "b5fa0942500f3afc0100a8556ca36a99"
server_key_version: "encryptionkey@0"
I20260620 20:16:19.443377 21891 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260620 20:16:19.447178 21905 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:16:19.448639 21891 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 20:16:19.448837 21891 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986536687152-17252-0/minicluster-data/master-0/data,/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986536687152-17252-0/minicluster-data/master-0/wal
uuid: "dbea80af3cbf4ca3915c6736ad52aafb"
format_stamp: "Formatted at 2026-06-20 20:16:19 on dist-test-slave-2hpg"
server_key: "7f6bffc79569d8e001936c5fd9d8c143"
server_key_iv: "b5fa0942500f3afc0100a8556ca36a99"
server_key_version: "encryptionkey@0"
I20260620 20:16:19.449079 21891 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986536687152-17252-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986536687152-17252-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986536687152-17252-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:16:19.455598 21891 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:16:19.456507 21891 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:16:19.456817 21891 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:16:19.474097 21891 rpc_server.cc:307] RPC server started. Bound to: 127.16.217.62:40285
I20260620 20:16:19.474217 21956 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.217.62:40285 every 8 connection(s)
I20260620 20:16:19.475504 21891 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986536687152-17252-0/minicluster-data/master-0/data/info.pb
I20260620 20:16:19.479094 17252 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu as pid 21891
I20260620 20:16:19.479300 17252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986536687152-17252-0/minicluster-data/master-0/wal/instance
I20260620 20:16:19.479521 21957 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:16:19.480129 17252 external_mini_cluster.cc:1428] Setting key 5541d5edbf43f2ca2bb94675f3f2eb69
I20260620 20:16:19.489523 21960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:19.482380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54654 (local address 127.16.217.62:40285)
0620 20:16:19.483629 (+ 1249us) server_negotiation.cc:206] Beginning negotiation
0620 20:16:19.483641 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 20:16:19.483703 (+ 62us) server_negotiation.cc:407] Connection header received
0620 20:16:19.485541 (+ 1838us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:16:19.485564 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:16:19.486090 (+ 526us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:16:19.486449 (+ 359us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:16:19.487574 (+ 1125us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:16:19.487581 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:16:19.487595 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:16:19.487622 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:16:19.487688 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:16:19.487708 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:16:19.487934 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0620 20:16:19.488047 (+ 113us) server_negotiation.cc:299] Negotiation successful
0620 20:16:19.488356 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":763,"thread_start_us":532,"threads_started":1}
I20260620 20:16:19.492941 21957 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dbea80af3cbf4ca3915c6736ad52aafb: Bootstrap starting.
I20260620 20:16:19.495793 21957 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dbea80af3cbf4ca3915c6736ad52aafb: Neither blocks nor log segments found. Creating new log.
I20260620 20:16:19.496770 21957 log.cc:824] T 00000000000000000000000000000000 P dbea80af3cbf4ca3915c6736ad52aafb: Log is configured to *not* fsync() on all Append() calls
I20260620 20:16:19.499433 21957 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dbea80af3cbf4ca3915c6736ad52aafb: No bootstrap required, opened a new log
I20260620 20:16:19.506402 21957 raft_consensus.cc:348] T 00000000000000000000000000000000 P dbea80af3cbf4ca3915c6736ad52aafb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbea80af3cbf4ca3915c6736ad52aafb" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 40285 } }
I20260620 20:16:19.506784 21957 raft_consensus.cc:374] T 00000000000000000000000000000000 P dbea80af3cbf4ca3915c6736ad52aafb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:16:19.506919 21957 raft_consensus.cc:729] T 00000000000000000000000000000000 P dbea80af3cbf4ca3915c6736ad52aafb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dbea80af3cbf4ca3915c6736ad52aafb, State: Initialized, Role: FOLLOWER
I20260620 20:16:19.507552 21957 consensus_queue.cc:260] T 00000000000000000000000000000000 P dbea80af3cbf4ca3915c6736ad52aafb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbea80af3cbf4ca3915c6736ad52aafb" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 40285 } }
I20260620 20:16:19.507757 21957 raft_consensus.cc:388] T 00000000000000000000000000000000 P dbea80af3cbf4ca3915c6736ad52aafb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 20:16:19.507853 21957 raft_consensus.cc:482] T 00000000000000000000000000000000 P dbea80af3cbf4ca3915c6736ad52aafb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 20:16:19.507967 21957 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dbea80af3cbf4ca3915c6736ad52aafb [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:16:19.510035 21957 raft_consensus.cc:504] T 00000000000000000000000000000000 P dbea80af3cbf4ca3915c6736ad52aafb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbea80af3cbf4ca3915c6736ad52aafb" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 40285 } }
I20260620 20:16:19.510470 21957 leader_election.cc:302] T 00000000000000000000000000000000 P dbea80af3cbf4ca3915c6736ad52aafb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dbea80af3cbf4ca3915c6736ad52aafb; no voters:
I20260620 20:16:19.511020 21957 leader_election.cc:288] T 00000000000000000000000000000000 P dbea80af3cbf4ca3915c6736ad52aafb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 20:16:19.511265 21962 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dbea80af3cbf4ca3915c6736ad52aafb [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:16:19.512070 21962 raft_consensus.cc:686] T 00000000000000000000000000000000 P dbea80af3cbf4ca3915c6736ad52aafb [term 1 LEADER]: Becoming Leader. State: Replica: dbea80af3cbf4ca3915c6736ad52aafb, State: Running, Role: LEADER
I20260620 20:16:19.512636 21962 consensus_queue.cc:237] T 00000000000000000000000000000000 P dbea80af3cbf4ca3915c6736ad52aafb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbea80af3cbf4ca3915c6736ad52aafb" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 40285 } }
I20260620 20:16:19.513089 21957 sys_catalog.cc:564] T 00000000000000000000000000000000 P dbea80af3cbf4ca3915c6736ad52aafb [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:16:19.516079 21964 sys_catalog.cc:455] T 00000000000000000000000000000000 P dbea80af3cbf4ca3915c6736ad52aafb [sys.catalog]: SysCatalogTable state changed. Reason: New leader dbea80af3cbf4ca3915c6736ad52aafb. Latest consensus state: current_term: 1 leader_uuid: "dbea80af3cbf4ca3915c6736ad52aafb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbea80af3cbf4ca3915c6736ad52aafb" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 40285 } } }
I20260620 20:16:19.516526 21964 sys_catalog.cc:458] T 00000000000000000000000000000000 P dbea80af3cbf4ca3915c6736ad52aafb [sys.catalog]: This master's current role is: LEADER
I20260620 20:16:19.516323 21963 sys_catalog.cc:455] T 00000000000000000000000000000000 P dbea80af3cbf4ca3915c6736ad52aafb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dbea80af3cbf4ca3915c6736ad52aafb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbea80af3cbf4ca3915c6736ad52aafb" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 40285 } } }
I20260620 20:16:19.516755 21963 sys_catalog.cc:458] T 00000000000000000000000000000000 P dbea80af3cbf4ca3915c6736ad52aafb [sys.catalog]: This master's current role is: LEADER
I20260620 20:16:19.519030 21968 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 20:16:19.525218 21968 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 20:16:19.534072 21968 catalog_manager.cc:1347] Generated new cluster ID: d45e296383c34356bdcb5adeb34c2382
I20260620 20:16:19.534210 21968 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 20:16:19.565104 21968 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 20:16:19.565622 21968 catalog_manager.cc:1504] Loading token signing keys...
I20260620 20:16:19.598281 21968 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dbea80af3cbf4ca3915c6736ad52aafb: Generated new TSK 0
I20260620 20:16:19.599197 21968 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 20:16:19.604272 17252 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
/tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986536687152-17252-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986536687152-17252-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986536687152-17252-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986536687152-17252-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.217.1:0
--local_ip_for_outbound_sockets=127.16.217.1
--webserver_interface=127.16.217.1
--webserver_port=0
--tserver_master_addrs=127.16.217.62:40285
--builtin_ntp_servers=127.16.217.20:46727
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260620 20:16:19.957161 21981 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:16:19.957428 21981 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:16:19.957605 21981 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:16:19.965008 21981 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:16:19.965116 21981 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:16:19.965292 21981 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.217.1
I20260620 20:16:19.974552 21981 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.217.20:46727
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986536687152-17252-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986536687152-17252-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.217.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986536687152-17252-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.217.1
--webserver_port=0
--tserver_master_addrs=127.16.217.62:40285
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.217.1
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986536687152-17252-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:16:19.975914 21981 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:16:19.977779 21981 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:16:19.991436 21987 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:16:19.991477 21986 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:16:19.992056 21989 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:16:19.992866 21981 server_base.cc:1034] running on GCE node
I20260620 20:16:19.993460 21981 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:16:19.994553 21981 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:16:19.995754 21981 hybrid_clock.cc:648] HybridClock initialized: now 1781986579995715 us; error 26 us; skew 500 ppm
I20260620 20:16:19.996115 21981 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 20:16:19.998521 21981 webserver.cc:460] Webserver started at http://127.16.217.1:39549/ using document root <none> and password file <none>
I20260620 20:16:19.999233 21981 fs_manager.cc:359] Metadata directory not provided
I20260620 20:16:19.999348 21981 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:16:19.999675 21981 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:16:20.002602 21981 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986536687152-17252-0/minicluster-data/ts-0/data/instance:
uuid: "af1184475ce44f2a8ceb6434ac7f13e4"
format_stamp: "Formatted at 2026-06-20 20:16:19 on dist-test-slave-2hpg"
server_key: "66154f8d33c13d8a6d237e243d58bc8c"
server_key_iv: "73c86bac08ee8ac7ed66c14a104c4266"
server_key_version: "encryptionkey@0"
I20260620 20:16:20.003382 21981 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986536687152-17252-0/minicluster-data/ts-0/wal/instance:
uuid: "af1184475ce44f2a8ceb6434ac7f13e4"
format_stamp: "Formatted at 2026-06-20 20:16:19 on dist-test-slave-2hpg"
server_key: "66154f8d33c13d8a6d237e243d58bc8c"
server_key_iv: "73c86bac08ee8ac7ed66c14a104c4266"
server_key_version: "encryptionkey@0"
I20260620 20:16:20.009058 21981 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 20:16:20.012828 21995 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:16:20.014587 21981 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 20:16:20.014780 21981 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986536687152-17252-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986536687152-17252-0/minicluster-data/ts-0/wal
uuid: "af1184475ce44f2a8ceb6434ac7f13e4"
format_stamp: "Formatted at 2026-06-20 20:16:19 on dist-test-slave-2hpg"
server_key: "66154f8d33c13d8a6d237e243d58bc8c"
server_key_iv: "73c86bac08ee8ac7ed66c14a104c4266"
server_key_version: "encryptionkey@0"
I20260620 20:16:20.015024 21981 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986536687152-17252-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986536687152-17252-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986536687152-17252-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:16:20.020902 21981 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:16:20.021746 21981 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:16:20.022056 21981 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:16:20.023044 21981 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:16:20.024511 21981 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:16:20.024631 21981 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:16:20.024749 21981 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:16:20.024861 21981 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:16:20.056951 21981 rpc_server.cc:307] RPC server started. Bound to: 127.16.217.1:39271
I20260620 20:16:20.057034 22107 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.217.1:39271 every 8 connection(s)
I20260620 20:16:20.058310 21981 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986536687152-17252-0/minicluster-data/ts-0/data/info.pb
I20260620 20:16:20.066157 22110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:20.060739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.217.1:56037 (local address 127.16.217.62:40285)
0620 20:16:20.061156 (+ 417us) server_negotiation.cc:206] Beginning negotiation
0620 20:16:20.061161 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:16:20.062283 (+ 1122us) server_negotiation.cc:407] Connection header received
0620 20:16:20.063965 (+ 1682us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:16:20.063970 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:16:20.064044 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:16:20.064157 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:16:20.065400 (+ 1243us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:16:20.065405 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:16:20.065409 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:16:20.065425 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:16:20.065469 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:16:20.065480 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:16:20.065610 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0620 20:16:20.065794 (+ 184us) server_negotiation.cc:299] Negotiation successful
0620 20:16:20.065862 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":319,"thread_start_us":222,"threads_started":1}
I20260620 20:16:20.066910 17252 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu as pid 21981
I20260620 20:16:20.067086 17252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986536687152-17252-0/minicluster-data/ts-0/wal/instance
I20260620 20:16:20.067147 22111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:20.061198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:40285 (local address 127.16.217.1:56037)
0620 20:16:20.062142 (+ 944us) negotiation.cc:107] Waiting for socket to connect
0620 20:16:20.062178 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 20:16:20.063690 (+ 1512us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:16:20.064381 (+ 691us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:16:20.064389 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:16:20.064485 (+ 96us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:16:20.065103 (+ 618us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 20:16:20.065105 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:16:20.065140 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 20:16:20.065220 (+ 80us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:16:20.065571 (+ 351us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:16:20.065581 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:16:20.065589 (+ 8us) client_negotiation.cc:769] Sending connection context
0620 20:16:20.065866 (+ 277us) client_negotiation.cc:240] Negotiation successful
0620 20:16:20.066155 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":275,"threads_started":1}
I20260620 20:16:20.067922 17252 external_mini_cluster.cc:1428] Setting key 4c3f65a719eb17a04709540e177296a6
I20260620 20:16:20.068727 22108 heartbeater.cc:347] Connected to a master server at 127.16.217.62:40285
I20260620 20:16:20.069028 22108 heartbeater.cc:464] Registering TS with master...
I20260620 20:16:20.069644 22108 heartbeater.cc:511] Master 127.16.217.62:40285 requested a full tablet report, sending...
I20260620 20:16:20.070736 17252 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
/tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986536687152-17252-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986536687152-17252-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986536687152-17252-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986536687152-17252-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.217.2:0
--local_ip_for_outbound_sockets=127.16.217.2
--webserver_interface=127.16.217.2
--webserver_port=0
--tserver_master_addrs=127.16.217.62:40285
--builtin_ntp_servers=127.16.217.20:46727
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260620 20:16:20.071909 21922 ts_manager.cc:194] Registered new tserver with Master: af1184475ce44f2a8ceb6434ac7f13e4 (127.16.217.1:39271)
W20260620 20:16:20.391361 22112 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:16:20.391690 22112 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:16:20.391851 22112 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:16:20.398977 22112 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:16:20.399116 22112 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:16:20.399289 22112 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.217.2
I20260620 20:16:20.408167 22112 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.217.20:46727
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986536687152-17252-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986536687152-17252-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.217.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986536687152-17252-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.217.2
--webserver_port=0
--tserver_master_addrs=127.16.217.62:40285
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.217.2
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986536687152-17252-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:16:20.409478 22112 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:16:20.411230 22112 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:16:20.424489 22117 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:16:20.424476 22118 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:16:20.424890 22120 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:16:20.425058 22112 server_base.cc:1034] running on GCE node
I20260620 20:16:20.425981 22112 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:16:20.426857 22112 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:16:20.428048 22112 hybrid_clock.cc:648] HybridClock initialized: now 1781986580427987 us; error 59 us; skew 500 ppm
I20260620 20:16:20.428360 22112 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 20:16:20.430620 22112 webserver.cc:460] Webserver started at http://127.16.217.2:40259/ using document root <none> and password file <none>
I20260620 20:16:20.431309 22112 fs_manager.cc:359] Metadata directory not provided
I20260620 20:16:20.431396 22112 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:16:20.431710 22112 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:16:20.434688 22112 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986536687152-17252-0/minicluster-data/ts-1/data/instance:
uuid: "975155771e5147ae9520308c6bb0ca26"
format_stamp: "Formatted at 2026-06-20 20:16:20 on dist-test-slave-2hpg"
server_key: "2d00138e1de0732c812f799a70ae9098"
server_key_iv: "a848a4c2e39e5fc95438ee7fa3ea2a91"
server_key_version: "encryptionkey@0"
I20260620 20:16:20.435487 22112 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986536687152-17252-0/minicluster-data/ts-1/wal/instance:
uuid: "975155771e5147ae9520308c6bb0ca26"
format_stamp: "Formatted at 2026-06-20 20:16:20 on dist-test-slave-2hpg"
server_key: "2d00138e1de0732c812f799a70ae9098"
server_key_iv: "a848a4c2e39e5fc95438ee7fa3ea2a91"
server_key_version: "encryptionkey@0"
I20260620 20:16:20.440625 22112 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 20:16:20.444094 22126 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:16:20.445390 22112 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 20:16:20.445559 22112 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986536687152-17252-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986536687152-17252-0/minicluster-data/ts-1/wal
uuid: "975155771e5147ae9520308c6bb0ca26"
format_stamp: "Formatted at 2026-06-20 20:16:20 on dist-test-slave-2hpg"
server_key: "2d00138e1de0732c812f799a70ae9098"
server_key_iv: "a848a4c2e39e5fc95438ee7fa3ea2a91"
server_key_version: "encryptionkey@0"
I20260620 20:16:20.445772 22112 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986536687152-17252-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986536687152-17252-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986536687152-17252-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:16:20.451383 22112 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:16:20.452162 22112 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:16:20.452423 22112 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:16:20.453326 22112 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:16:20.454844 22112 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:16:20.454974 22112 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:16:20.455094 22112 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:16:20.455178 22112 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:16:20.488683 22112 rpc_server.cc:307] RPC server started. Bound to: 127.16.217.2:44253
I20260620 20:16:20.488780 22238 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.217.2:44253 every 8 connection(s)
I20260620 20:16:20.490159 22112 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986536687152-17252-0/minicluster-data/ts-1/data/info.pb
I20260620 20:16:20.498389 22110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:20.493073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.217.2:56687 (local address 127.16.217.62:40285)
0620 20:16:20.493273 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0620 20:16:20.493280 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:16:20.494700 (+ 1420us) server_negotiation.cc:407] Connection header received
0620 20:16:20.496314 (+ 1614us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:16:20.496317 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:16:20.496368 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:16:20.496453 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:16:20.497705 (+ 1252us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:16:20.497709 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:16:20.497711 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:16:20.497724 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:16:20.497751 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:16:20.497762 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:16:20.497849 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0620 20:16:20.498068 (+ 219us) server_negotiation.cc:299] Negotiation successful
0620 20:16:20.498144 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 20:16:20.499284 22241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:20.493646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:40285 (local address 127.16.217.2:56687)
0620 20:16:20.494549 (+ 903us) negotiation.cc:107] Waiting for socket to connect
0620 20:16:20.494588 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 20:16:20.496082 (+ 1494us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:16:20.496667 (+ 585us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:16:20.496674 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:16:20.496767 (+ 93us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:16:20.497441 (+ 674us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 20:16:20.497443 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:16:20.497479 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 20:16:20.497563 (+ 84us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:16:20.497842 (+ 279us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:16:20.497849 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:16:20.497854 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 20:16:20.498137 (+ 283us) client_negotiation.cc:240] Negotiation successful
0620 20:16:20.498363 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":459,"thread_start_us":288,"threads_started":1}
I20260620 20:16:20.500173 17252 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu as pid 22112
I20260620 20:16:20.500342 17252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986536687152-17252-0/minicluster-data/ts-1/wal/instance
I20260620 20:16:20.500802 22239 heartbeater.cc:347] Connected to a master server at 127.16.217.62:40285
I20260620 20:16:20.500914 17252 external_mini_cluster.cc:1428] Setting key 072a39a437ca5906ab0553b05a84bab2
I20260620 20:16:20.501122 22239 heartbeater.cc:464] Registering TS with master...
I20260620 20:16:20.501761 22239 heartbeater.cc:511] Master 127.16.217.62:40285 requested a full tablet report, sending...
I20260620 20:16:20.503290 21922 ts_manager.cc:194] Registered new tserver with Master: 975155771e5147ae9520308c6bb0ca26 (127.16.217.2:44253)
I20260620 20:16:20.503868 17252 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
/tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986536687152-17252-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986536687152-17252-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986536687152-17252-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986536687152-17252-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.217.3:0
--local_ip_for_outbound_sockets=127.16.217.3
--webserver_interface=127.16.217.3
--webserver_port=0
--tserver_master_addrs=127.16.217.62:40285
--builtin_ntp_servers=127.16.217.20:46727
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260620 20:16:20.832913 22242 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:16:20.833189 22242 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:16:20.833369 22242 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:16:20.840660 22242 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:16:20.840763 22242 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:16:20.840962 22242 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.217.3
I20260620 20:16:20.850304 22242 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.217.20:46727
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986536687152-17252-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986536687152-17252-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.217.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986536687152-17252-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.217.3
--webserver_port=0
--tserver_master_addrs=127.16.217.62:40285
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.217.3
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986536687152-17252-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:16:20.851591 22242 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:16:20.853427 22242 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:16:20.866107 22250 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:16:20.866190 22247 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:16:20.866168 22248 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:16:20.867609 22242 server_base.cc:1034] running on GCE node
I20260620 20:16:20.868296 22242 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:16:20.869375 22242 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:16:20.870631 22242 hybrid_clock.cc:648] HybridClock initialized: now 1781986580870598 us; error 80 us; skew 500 ppm
I20260620 20:16:20.870994 22242 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 20:16:20.873318 22242 webserver.cc:460] Webserver started at http://127.16.217.3:37479/ using document root <none> and password file <none>
I20260620 20:16:20.874084 22242 fs_manager.cc:359] Metadata directory not provided
I20260620 20:16:20.874207 22242 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:16:20.874527 22242 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:16:20.877311 22242 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986536687152-17252-0/minicluster-data/ts-2/data/instance:
uuid: "8485cedc3edc49769c593eacfa36b245"
format_stamp: "Formatted at 2026-06-20 20:16:20 on dist-test-slave-2hpg"
server_key: "ab86fa5f291495845bf0d082e2fe04cf"
server_key_iv: "3d5fdb0e7ee6117e37c828e50443a62e"
server_key_version: "encryptionkey@0"
I20260620 20:16:20.878193 22242 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986536687152-17252-0/minicluster-data/ts-2/wal/instance:
uuid: "8485cedc3edc49769c593eacfa36b245"
format_stamp: "Formatted at 2026-06-20 20:16:20 on dist-test-slave-2hpg"
server_key: "ab86fa5f291495845bf0d082e2fe04cf"
server_key_iv: "3d5fdb0e7ee6117e37c828e50443a62e"
server_key_version: "encryptionkey@0"
I20260620 20:16:20.883582 22242 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260620 20:16:20.887339 22256 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:16:20.888849 22242 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 20:16:20.889041 22242 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986536687152-17252-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986536687152-17252-0/minicluster-data/ts-2/wal
uuid: "8485cedc3edc49769c593eacfa36b245"
format_stamp: "Formatted at 2026-06-20 20:16:20 on dist-test-slave-2hpg"
server_key: "ab86fa5f291495845bf0d082e2fe04cf"
server_key_iv: "3d5fdb0e7ee6117e37c828e50443a62e"
server_key_version: "encryptionkey@0"
I20260620 20:16:20.889284 22242 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986536687152-17252-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986536687152-17252-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986536687152-17252-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:16:20.895473 22242 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:16:20.896333 22242 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:16:20.896606 22242 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:16:20.897547 22242 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:16:20.899058 22242 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:16:20.899194 22242 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:16:20.899327 22242 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:16:20.899413 22242 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:16:20.935122 22242 rpc_server.cc:307] RPC server started. Bound to: 127.16.217.3:36545
I20260620 20:16:20.935177 22368 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.217.3:36545 every 8 connection(s)
I20260620 20:16:20.936484 22242 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986536687152-17252-0/minicluster-data/ts-2/data/info.pb
I20260620 20:16:20.942560 17252 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu as pid 22242
I20260620 20:16:20.942719 17252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986536687152-17252-0/minicluster-data/ts-2/wal/instance
I20260620 20:16:20.943172 17252 external_mini_cluster.cc:1428] Setting key 81acd075033ebfae71dafaa8c8d42ee5
I20260620 20:16:20.945039 22110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:20.939114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.217.3:32891 (local address 127.16.217.62:40285)
0620 20:16:20.939334 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0620 20:16:20.939349 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 20:16:20.940807 (+ 1458us) server_negotiation.cc:407] Connection header received
0620 20:16:20.942570 (+ 1763us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:16:20.942577 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:16:20.942658 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:16:20.942781 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:16:20.944229 (+ 1448us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:16:20.944233 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:16:20.944237 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:16:20.944251 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:16:20.944280 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:16:20.944292 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:16:20.944406 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0620 20:16:20.944648 (+ 242us) server_negotiation.cc:299] Negotiation successful
0620 20:16:20.944709 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 20:16:20.946056 22371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:20.939724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:40285 (local address 127.16.217.3:32891)
0620 20:16:20.940650 (+ 926us) negotiation.cc:107] Waiting for socket to connect
0620 20:16:20.940691 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 20:16:20.942304 (+ 1613us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:16:20.943015 (+ 711us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:16:20.943024 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:16:20.943115 (+ 91us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:16:20.943912 (+ 797us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 20:16:20.943917 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:16:20.943969 (+ 52us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 20:16:20.944076 (+ 107us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:16:20.944391 (+ 315us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:16:20.944402 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:16:20.944410 (+ 8us) client_negotiation.cc:769] Sending connection context
0620 20:16:20.944701 (+ 291us) client_negotiation.cc:240] Negotiation successful
0620 20:16:20.944911 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":468,"thread_start_us":280,"threads_started":1}
I20260620 20:16:20.947274 22369 heartbeater.cc:347] Connected to a master server at 127.16.217.62:40285
I20260620 20:16:20.947515 22369 heartbeater.cc:464] Registering TS with master...
I20260620 20:16:20.948045 22369 heartbeater.cc:511] Master 127.16.217.62:40285 requested a full tablet report, sending...
I20260620 20:16:20.949277 21922 ts_manager.cc:194] Registered new tserver with Master: 8485cedc3edc49769c593eacfa36b245 (127.16.217.3:36545)
I20260620 20:16:20.958426 17252 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 20:16:20.965372 22110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:20.962925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59902 (local address 127.16.217.62:40285)
0620 20:16:20.963074 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0620 20:16:20.963078 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:16:20.963612 (+ 534us) server_negotiation.cc:407] Connection header received
0620 20:16:20.963741 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:16:20.963745 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:16:20.963827 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:16:20.963913 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:16:20.964881 (+ 968us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:16:20.964885 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:16:20.964888 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:16:20.964902 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:16:20.964930 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:16:20.964941 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:16:20.965051 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0620 20:16:20.965119 (+ 68us) server_negotiation.cc:299] Negotiation successful
0620 20:16:20.965161 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 20:16:20.969424 21922 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59902:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 20:16:20.976691 21922 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 20:16:20.995791 22381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:20.989751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32778 (local address 127.16.217.1:39271)
0620 20:16:20.992423 (+ 2672us) server_negotiation.cc:206] Beginning negotiation
0620 20:16:20.992434 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 20:16:20.992467 (+ 33us) server_negotiation.cc:407] Connection header received
0620 20:16:20.992538 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:16:20.992546 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:16:20.992835 (+ 289us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:16:20.993567 (+ 732us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:16:20.994874 (+ 1307us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:16:20.994883 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:16:20.994898 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:16:20.994927 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:16:20.994989 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:16:20.995011 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:16:20.995156 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0620 20:16:20.995304 (+ 148us) server_negotiation.cc:299] Negotiation successful
0620 20:16:20.995401 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2513,"thread_start_us":2399,"threads_started":1}
I20260620 20:16:20.995862 22382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:20.989726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.1:39271 (local address 127.0.0.1:32778)
0620 20:16:20.991364 (+ 1638us) negotiation.cc:107] Waiting for socket to connect
0620 20:16:20.991380 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 20:16:20.991463 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:16:20.993029 (+ 1566us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:16:20.993036 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:16:20.993204 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:16:20.994370 (+ 1166us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 20:16:20.994376 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:16:20.994395 (+ 19us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 20:16:20.994447 (+ 52us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:16:20.995124 (+ 677us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:16:20.995136 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:16:20.995145 (+ 9us) client_negotiation.cc:769] Sending connection context
0620 20:16:20.995319 (+ 174us) client_negotiation.cc:240] Negotiation successful
0620 20:16:20.995419 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1515,"thread_start_us":1390,"threads_started":1}
I20260620 20:16:20.996719 22384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:20.989818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56458 (local address 127.16.217.3:36545)
0620 20:16:20.992007 (+ 2189us) server_negotiation.cc:206] Beginning negotiation
0620 20:16:20.992015 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 20:16:20.992050 (+ 35us) server_negotiation.cc:407] Connection header received
0620 20:16:20.992126 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:16:20.992136 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:16:20.992891 (+ 755us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:16:20.993976 (+ 1085us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:16:20.995543 (+ 1567us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:16:20.995551 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:16:20.995563 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:16:20.995581 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:16:20.995626 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:16:20.995642 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:16:20.996191 (+ 549us) server_negotiation.cc:1035] Waiting for connection context
0620 20:16:20.996260 (+ 69us) server_negotiation.cc:299] Negotiation successful
0620 20:16:20.996354 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2007,"thread_start_us":1930,"threads_started":1}
I20260620 20:16:20.996886 22383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:20.989541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.3:36545 (local address 127.0.0.1:56458)
0620 20:16:20.990700 (+ 1159us) negotiation.cc:107] Waiting for socket to connect
0620 20:16:20.990741 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 20:16:20.990866 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:16:20.993579 (+ 2713us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:16:20.993584 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:16:20.993606 (+ 22us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:16:20.994292 (+ 686us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 20:16:20.994295 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:16:20.994330 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 20:16:20.994421 (+ 91us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:16:20.995770 (+ 1349us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:16:20.995774 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:16:20.995811 (+ 37us) client_negotiation.cc:769] Sending connection context
0620 20:16:20.996393 (+ 582us) client_negotiation.cc:240] Negotiation successful
0620 20:16:20.996442 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":930,"thread_start_us":780,"threads_started":1}
I20260620 20:16:20.997557 22385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:20.991725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.2:44253 (local address 127.0.0.1:58372)
0620 20:16:20.992683 (+ 958us) negotiation.cc:107] Waiting for socket to connect
0620 20:16:20.992698 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 20:16:20.992793 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:16:20.994108 (+ 1315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:16:20.994114 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:16:20.994144 (+ 30us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:16:20.994821 (+ 677us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 20:16:20.994827 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:16:20.994846 (+ 19us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 20:16:20.994888 (+ 42us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:16:20.995798 (+ 910us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:16:20.995804 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:16:20.995812 (+ 8us) client_negotiation.cc:769] Sending connection context
0620 20:16:20.995952 (+ 140us) client_negotiation.cc:240] Negotiation successful
0620 20:16:20.996005 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":854,"thread_start_us":717,"threads_started":1}
I20260620 20:16:20.998206 22386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:20.992263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58372 (local address 127.16.217.2:44253)
0620 20:16:20.993574 (+ 1311us) server_negotiation.cc:206] Beginning negotiation
0620 20:16:20.993587 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 20:16:20.993615 (+ 28us) server_negotiation.cc:407] Connection header received
0620 20:16:20.993681 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:16:20.993689 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:16:20.993927 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:16:20.994571 (+ 644us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:16:20.995556 (+ 985us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:16:20.995563 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:16:20.995577 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:16:20.995597 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:16:20.995642 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:16:20.995659 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:16:20.997043 (+ 1384us) server_negotiation.cc:1035] Waiting for connection context
0620 20:16:20.997850 (+ 807us) server_negotiation.cc:299] Negotiation successful
0620 20:16:20.997969 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1146,"thread_start_us":931,"threads_started":1}
I20260620 20:16:21.000289 22304 tablet_service.cc:1467] Processing CreateTablet for tablet 2fb8429558a7462c8f4de4180481b761 (DEFAULT_TABLE table=test-table [id=34a734114c744e10b929f4d36d5bab3f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:16:21.000111 22043 tablet_service.cc:1467] Processing CreateTablet for tablet 2fb8429558a7462c8f4de4180481b761 (DEFAULT_TABLE table=test-table [id=34a734114c744e10b929f4d36d5bab3f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:16:21.001245 22174 tablet_service.cc:1467] Processing CreateTablet for tablet 2fb8429558a7462c8f4de4180481b761 (DEFAULT_TABLE table=test-table [id=34a734114c744e10b929f4d36d5bab3f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:16:21.001754 22043 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2fb8429558a7462c8f4de4180481b761. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:16:21.001750 22304 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2fb8429558a7462c8f4de4180481b761. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:16:21.002270 22174 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2fb8429558a7462c8f4de4180481b761. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:16:21.013947 22387 tablet_bootstrap.cc:492] T 2fb8429558a7462c8f4de4180481b761 P 975155771e5147ae9520308c6bb0ca26: Bootstrap starting.
I20260620 20:16:21.016525 22388 tablet_bootstrap.cc:492] T 2fb8429558a7462c8f4de4180481b761 P af1184475ce44f2a8ceb6434ac7f13e4: Bootstrap starting.
I20260620 20:16:21.017299 22389 tablet_bootstrap.cc:492] T 2fb8429558a7462c8f4de4180481b761 P 8485cedc3edc49769c593eacfa36b245: Bootstrap starting.
I20260620 20:16:21.017792 22387 tablet_bootstrap.cc:654] T 2fb8429558a7462c8f4de4180481b761 P 975155771e5147ae9520308c6bb0ca26: Neither blocks nor log segments found. Creating new log.
I20260620 20:16:21.019917 22387 log.cc:824] T 2fb8429558a7462c8f4de4180481b761 P 975155771e5147ae9520308c6bb0ca26: Log is configured to *not* fsync() on all Append() calls
I20260620 20:16:21.021008 22388 tablet_bootstrap.cc:654] T 2fb8429558a7462c8f4de4180481b761 P af1184475ce44f2a8ceb6434ac7f13e4: Neither blocks nor log segments found. Creating new log.
I20260620 20:16:21.021682 22389 tablet_bootstrap.cc:654] T 2fb8429558a7462c8f4de4180481b761 P 8485cedc3edc49769c593eacfa36b245: Neither blocks nor log segments found. Creating new log.
I20260620 20:16:21.022068 22388 log.cc:824] T 2fb8429558a7462c8f4de4180481b761 P af1184475ce44f2a8ceb6434ac7f13e4: Log is configured to *not* fsync() on all Append() calls
I20260620 20:16:21.022835 22389 log.cc:824] T 2fb8429558a7462c8f4de4180481b761 P 8485cedc3edc49769c593eacfa36b245: Log is configured to *not* fsync() on all Append() calls
I20260620 20:16:21.023095 22387 tablet_bootstrap.cc:492] T 2fb8429558a7462c8f4de4180481b761 P 975155771e5147ae9520308c6bb0ca26: No bootstrap required, opened a new log
I20260620 20:16:21.023396 22387 ts_tablet_manager.cc:1397] T 2fb8429558a7462c8f4de4180481b761 P 975155771e5147ae9520308c6bb0ca26: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260620 20:16:21.024660 22388 tablet_bootstrap.cc:492] T 2fb8429558a7462c8f4de4180481b761 P af1184475ce44f2a8ceb6434ac7f13e4: No bootstrap required, opened a new log
I20260620 20:16:21.024919 22388 ts_tablet_manager.cc:1397] T 2fb8429558a7462c8f4de4180481b761 P af1184475ce44f2a8ceb6434ac7f13e4: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.003s
I20260620 20:16:21.025595 22389 tablet_bootstrap.cc:492] T 2fb8429558a7462c8f4de4180481b761 P 8485cedc3edc49769c593eacfa36b245: No bootstrap required, opened a new log
I20260620 20:16:21.025835 22389 ts_tablet_manager.cc:1397] T 2fb8429558a7462c8f4de4180481b761 P 8485cedc3edc49769c593eacfa36b245: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260620 20:16:21.032173 22388 raft_consensus.cc:348] T 2fb8429558a7462c8f4de4180481b761 P af1184475ce44f2a8ceb6434ac7f13e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8485cedc3edc49769c593eacfa36b245" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 36545 } } peers { permanent_uuid: "af1184475ce44f2a8ceb6434ac7f13e4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 39271 } } peers { permanent_uuid: "975155771e5147ae9520308c6bb0ca26" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 44253 } }
I20260620 20:16:21.032565 22388 raft_consensus.cc:374] T 2fb8429558a7462c8f4de4180481b761 P af1184475ce44f2a8ceb6434ac7f13e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:16:21.032692 22388 raft_consensus.cc:729] T 2fb8429558a7462c8f4de4180481b761 P af1184475ce44f2a8ceb6434ac7f13e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af1184475ce44f2a8ceb6434ac7f13e4, State: Initialized, Role: FOLLOWER
I20260620 20:16:21.033324 22388 consensus_queue.cc:260] T 2fb8429558a7462c8f4de4180481b761 P af1184475ce44f2a8ceb6434ac7f13e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8485cedc3edc49769c593eacfa36b245" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 36545 } } peers { permanent_uuid: "af1184475ce44f2a8ceb6434ac7f13e4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 39271 } } peers { permanent_uuid: "975155771e5147ae9520308c6bb0ca26" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 44253 } }
I20260620 20:16:21.033851 22387 raft_consensus.cc:348] T 2fb8429558a7462c8f4de4180481b761 P 975155771e5147ae9520308c6bb0ca26 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8485cedc3edc49769c593eacfa36b245" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 36545 } } peers { permanent_uuid: "af1184475ce44f2a8ceb6434ac7f13e4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 39271 } } peers { permanent_uuid: "975155771e5147ae9520308c6bb0ca26" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 44253 } }
I20260620 20:16:21.034368 22387 raft_consensus.cc:374] T 2fb8429558a7462c8f4de4180481b761 P 975155771e5147ae9520308c6bb0ca26 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:16:21.034431 22108 heartbeater.cc:503] Master 127.16.217.62:40285 was elected leader, sending a full tablet report...
I20260620 20:16:21.034474 22387 raft_consensus.cc:729] T 2fb8429558a7462c8f4de4180481b761 P 975155771e5147ae9520308c6bb0ca26 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 975155771e5147ae9520308c6bb0ca26, State: Initialized, Role: FOLLOWER
I20260620 20:16:21.035086 22387 consensus_queue.cc:260] T 2fb8429558a7462c8f4de4180481b761 P 975155771e5147ae9520308c6bb0ca26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8485cedc3edc49769c593eacfa36b245" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 36545 } } peers { permanent_uuid: "af1184475ce44f2a8ceb6434ac7f13e4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 39271 } } peers { permanent_uuid: "975155771e5147ae9520308c6bb0ca26" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 44253 } }
I20260620 20:16:21.036201 22239 heartbeater.cc:503] Master 127.16.217.62:40285 was elected leader, sending a full tablet report...
I20260620 20:16:21.036095 22389 raft_consensus.cc:348] T 2fb8429558a7462c8f4de4180481b761 P 8485cedc3edc49769c593eacfa36b245 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8485cedc3edc49769c593eacfa36b245" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 36545 } } peers { permanent_uuid: "af1184475ce44f2a8ceb6434ac7f13e4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 39271 } } peers { permanent_uuid: "975155771e5147ae9520308c6bb0ca26" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 44253 } }
I20260620 20:16:21.036484 22389 raft_consensus.cc:374] T 2fb8429558a7462c8f4de4180481b761 P 8485cedc3edc49769c593eacfa36b245 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:16:21.036584 22389 raft_consensus.cc:729] T 2fb8429558a7462c8f4de4180481b761 P 8485cedc3edc49769c593eacfa36b245 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8485cedc3edc49769c593eacfa36b245, State: Initialized, Role: FOLLOWER
I20260620 20:16:21.037101 22389 consensus_queue.cc:260] T 2fb8429558a7462c8f4de4180481b761 P 8485cedc3edc49769c593eacfa36b245 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8485cedc3edc49769c593eacfa36b245" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 36545 } } peers { permanent_uuid: "af1184475ce44f2a8ceb6434ac7f13e4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 39271 } } peers { permanent_uuid: "975155771e5147ae9520308c6bb0ca26" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 44253 } }
I20260620 20:16:21.038200 22387 ts_tablet_manager.cc:1428] T 2fb8429558a7462c8f4de4180481b761 P 975155771e5147ae9520308c6bb0ca26: Time spent starting tablet: real 0.015s user 0.013s sys 0.000s
I20260620 20:16:21.039192 22369 heartbeater.cc:503] Master 127.16.217.62:40285 was elected leader, sending a full tablet report...
I20260620 20:16:21.039799 22389 ts_tablet_manager.cc:1428] T 2fb8429558a7462c8f4de4180481b761 P 8485cedc3edc49769c593eacfa36b245: Time spent starting tablet: real 0.014s user 0.008s sys 0.005s
I20260620 20:16:21.039783 22388 ts_tablet_manager.cc:1428] T 2fb8429558a7462c8f4de4180481b761 P af1184475ce44f2a8ceb6434ac7f13e4: Time spent starting tablet: real 0.015s user 0.008s sys 0.003s
W20260620 20:16:21.060464 22109 tablet.cc:2369] T 2fb8429558a7462c8f4de4180481b761 P af1184475ce44f2a8ceb6434ac7f13e4: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 20:16:21.188165 22370 tablet.cc:2369] T 2fb8429558a7462c8f4de4180481b761 P 8485cedc3edc49769c593eacfa36b245: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 20:16:21.219957 22395 raft_consensus.cc:482] T 2fb8429558a7462c8f4de4180481b761 P 8485cedc3edc49769c593eacfa36b245 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 20:16:21.220253 22395 raft_consensus.cc:504] T 2fb8429558a7462c8f4de4180481b761 P 8485cedc3edc49769c593eacfa36b245 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8485cedc3edc49769c593eacfa36b245" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 36545 } } peers { permanent_uuid: "af1184475ce44f2a8ceb6434ac7f13e4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 39271 } } peers { permanent_uuid: "975155771e5147ae9520308c6bb0ca26" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 44253 } }
I20260620 20:16:21.221997 22395 leader_election.cc:288] T 2fb8429558a7462c8f4de4180481b761 P 8485cedc3edc49769c593eacfa36b245 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af1184475ce44f2a8ceb6434ac7f13e4 (127.16.217.1:39271), 975155771e5147ae9520308c6bb0ca26 (127.16.217.2:44253)
I20260620 20:16:21.224877 22371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:21.222318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.2:44253 (local address 127.16.217.3:57999)
0620 20:16:21.222492 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0620 20:16:21.222513 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 20:16:21.222657 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:16:21.223219 (+ 562us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:16:21.223229 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:16:21.223258 (+ 29us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:16:21.224009 (+ 751us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 20:16:21.224011 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:16:21.224025 (+ 14us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 20:16:21.224064 (+ 39us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:16:21.224424 (+ 360us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:16:21.224431 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:16:21.224437 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 20:16:21.224548 (+ 111us) client_negotiation.cc:240] Negotiation successful
0620 20:16:21.224603 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53}
I20260620 20:16:21.224953 22386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:21.222589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.217.3:57999 (local address 127.16.217.2:44253)
0620 20:16:21.222780 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0620 20:16:21.222795 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 20:16:21.222818 (+ 23us) server_negotiation.cc:407] Connection header received
0620 20:16:21.222888 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:16:21.222896 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:16:21.222977 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:16:21.223133 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:16:21.224216 (+ 1083us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:16:21.224221 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:16:21.224226 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:16:21.224251 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:16:21.224320 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:16:21.224335 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:16:21.224455 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0620 20:16:21.224610 (+ 155us) server_negotiation.cc:299] Negotiation successful
0620 20:16:21.224669 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 20:16:21.225755 22381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:21.222524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.217.3:38933 (local address 127.16.217.1:39271)
0620 20:16:21.222718 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0620 20:16:21.222725 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:16:21.223662 (+ 937us) server_negotiation.cc:407] Connection header received
0620 20:16:21.223839 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:16:21.223846 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:16:21.223901 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:16:21.223997 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:16:21.225022 (+ 1025us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:16:21.225029 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:16:21.225035 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:16:21.225054 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:16:21.225091 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:16:21.225109 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:16:21.225202 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0620 20:16:21.225436 (+ 234us) server_negotiation.cc:299] Negotiation successful
0620 20:16:21.225490 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 20:16:21.225941 22396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:21.222338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.1:39271 (local address 127.16.217.3:38933)
0620 20:16:21.223579 (+ 1241us) negotiation.cc:107] Waiting for socket to connect
0620 20:16:21.223597 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 20:16:21.223700 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:16:21.224016 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:16:21.224021 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:16:21.224042 (+ 21us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:16:21.224802 (+ 760us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 20:16:21.224806 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:16:21.224821 (+ 15us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 20:16:21.224860 (+ 39us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:16:21.225305 (+ 445us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:16:21.225310 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:16:21.225314 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 20:16:21.225426 (+ 112us) client_negotiation.cc:240] Negotiation successful
0620 20:16:21.225476 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1102,"mutex_wait_us":319,"thread_start_us":449,"threads_started":1}
I20260620 20:16:21.226084 22194 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2fb8429558a7462c8f4de4180481b761" candidate_uuid: "8485cedc3edc49769c593eacfa36b245" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "975155771e5147ae9520308c6bb0ca26" is_pre_election: true
I20260620 20:16:21.226644 22194 raft_consensus.cc:2454] T 2fb8429558a7462c8f4de4180481b761 P 975155771e5147ae9520308c6bb0ca26 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8485cedc3edc49769c593eacfa36b245 in term 0.
I20260620 20:16:21.226854 22063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2fb8429558a7462c8f4de4180481b761" candidate_uuid: "8485cedc3edc49769c593eacfa36b245" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af1184475ce44f2a8ceb6434ac7f13e4" is_pre_election: true
I20260620 20:16:21.227339 22260 leader_election.cc:302] T 2fb8429558a7462c8f4de4180481b761 P 8485cedc3edc49769c593eacfa36b245 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8485cedc3edc49769c593eacfa36b245, 975155771e5147ae9520308c6bb0ca26; no voters:
I20260620 20:16:21.227362 22063 raft_consensus.cc:2454] T 2fb8429558a7462c8f4de4180481b761 P af1184475ce44f2a8ceb6434ac7f13e4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8485cedc3edc49769c593eacfa36b245 in term 0.
I20260620 20:16:21.227706 22395 raft_consensus.cc:2781] T 2fb8429558a7462c8f4de4180481b761 P 8485cedc3edc49769c593eacfa36b245 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 20:16:21.227909 22395 raft_consensus.cc:482] T 2fb8429558a7462c8f4de4180481b761 P 8485cedc3edc49769c593eacfa36b245 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 20:16:21.228029 22395 raft_consensus.cc:3037] T 2fb8429558a7462c8f4de4180481b761 P 8485cedc3edc49769c593eacfa36b245 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:16:21.230301 22395 raft_consensus.cc:504] T 2fb8429558a7462c8f4de4180481b761 P 8485cedc3edc49769c593eacfa36b245 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8485cedc3edc49769c593eacfa36b245" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 36545 } } peers { permanent_uuid: "af1184475ce44f2a8ceb6434ac7f13e4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 39271 } } peers { permanent_uuid: "975155771e5147ae9520308c6bb0ca26" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 44253 } }
I20260620 20:16:21.230939 22395 leader_election.cc:288] T 2fb8429558a7462c8f4de4180481b761 P 8485cedc3edc49769c593eacfa36b245 [CANDIDATE]: Term 1 election: Requested vote from peers af1184475ce44f2a8ceb6434ac7f13e4 (127.16.217.1:39271), 975155771e5147ae9520308c6bb0ca26 (127.16.217.2:44253)
I20260620 20:16:21.231436 22194 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2fb8429558a7462c8f4de4180481b761" candidate_uuid: "8485cedc3edc49769c593eacfa36b245" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "975155771e5147ae9520308c6bb0ca26"
I20260620 20:16:21.231436 22063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2fb8429558a7462c8f4de4180481b761" candidate_uuid: "8485cedc3edc49769c593eacfa36b245" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af1184475ce44f2a8ceb6434ac7f13e4"
I20260620 20:16:21.231659 22063 raft_consensus.cc:3037] T 2fb8429558a7462c8f4de4180481b761 P af1184475ce44f2a8ceb6434ac7f13e4 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:16:21.231660 22194 raft_consensus.cc:3037] T 2fb8429558a7462c8f4de4180481b761 P 975155771e5147ae9520308c6bb0ca26 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:16:21.234938 22194 raft_consensus.cc:2454] T 2fb8429558a7462c8f4de4180481b761 P 975155771e5147ae9520308c6bb0ca26 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8485cedc3edc49769c593eacfa36b245 in term 1.
I20260620 20:16:21.234938 22063 raft_consensus.cc:2454] T 2fb8429558a7462c8f4de4180481b761 P af1184475ce44f2a8ceb6434ac7f13e4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8485cedc3edc49769c593eacfa36b245 in term 1.
I20260620 20:16:21.235447 22259 leader_election.cc:302] T 2fb8429558a7462c8f4de4180481b761 P 8485cedc3edc49769c593eacfa36b245 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8485cedc3edc49769c593eacfa36b245, af1184475ce44f2a8ceb6434ac7f13e4; no voters:
I20260620 20:16:21.235793 22395 raft_consensus.cc:2781] T 2fb8429558a7462c8f4de4180481b761 P 8485cedc3edc49769c593eacfa36b245 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:16:21.236313 22395 raft_consensus.cc:686] T 2fb8429558a7462c8f4de4180481b761 P 8485cedc3edc49769c593eacfa36b245 [term 1 LEADER]: Becoming Leader. State: Replica: 8485cedc3edc49769c593eacfa36b245, State: Running, Role: LEADER
I20260620 20:16:21.236891 22395 consensus_queue.cc:237] T 2fb8429558a7462c8f4de4180481b761 P 8485cedc3edc49769c593eacfa36b245 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8485cedc3edc49769c593eacfa36b245" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 36545 } } peers { permanent_uuid: "af1184475ce44f2a8ceb6434ac7f13e4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 39271 } } peers { permanent_uuid: "975155771e5147ae9520308c6bb0ca26" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 44253 } }
I20260620 20:16:21.241618 21920 catalog_manager.cc:5495] T 2fb8429558a7462c8f4de4180481b761 P 8485cedc3edc49769c593eacfa36b245 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8485cedc3edc49769c593eacfa36b245 (127.16.217.3). New cstate: current_term: 1 leader_uuid: "8485cedc3edc49769c593eacfa36b245" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8485cedc3edc49769c593eacfa36b245" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 36545 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af1184475ce44f2a8ceb6434ac7f13e4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 39271 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "975155771e5147ae9520308c6bb0ca26" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 44253 } health_report { overall_health: UNKNOWN } } }
W20260620 20:16:21.242164 22240 tablet.cc:2369] T 2fb8429558a7462c8f4de4180481b761 P 975155771e5147ae9520308c6bb0ca26: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 20:16:21.291776 22384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:21.289839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56474 (local address 127.16.217.3:36545)
0620 20:16:21.290033 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0620 20:16:21.290038 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:16:21.290055 (+ 17us) server_negotiation.cc:407] Connection header received
0620 20:16:21.290211 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:16:21.290215 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:16:21.290293 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:16:21.290382 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:16:21.291254 (+ 872us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:16:21.291258 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:16:21.291261 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:16:21.291274 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:16:21.291303 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:16:21.291313 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:16:21.291413 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0620 20:16:21.291547 (+ 134us) server_negotiation.cc:299] Negotiation successful
0620 20:16:21.291590 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260620 20:16:21.300166 22194 raft_consensus.cc:1261] T 2fb8429558a7462c8f4de4180481b761 P 975155771e5147ae9520308c6bb0ca26 [term 1 FOLLOWER]: Refusing update from remote peer 8485cedc3edc49769c593eacfa36b245: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:16:21.300166 22063 raft_consensus.cc:1261] T 2fb8429558a7462c8f4de4180481b761 P af1184475ce44f2a8ceb6434ac7f13e4 [term 1 FOLLOWER]: Refusing update from remote peer 8485cedc3edc49769c593eacfa36b245: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:16:21.301131 22397 consensus_queue.cc:1035] T 2fb8429558a7462c8f4de4180481b761 P 8485cedc3edc49769c593eacfa36b245 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af1184475ce44f2a8ceb6434ac7f13e4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 39271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:16:21.301515 22395 consensus_queue.cc:1035] T 2fb8429558a7462c8f4de4180481b761 P 8485cedc3edc49769c593eacfa36b245 [LEADER]: Connected to new peer: Peer: permanent_uuid: "975155771e5147ae9520308c6bb0ca26" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 44253 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:16:21.315649 22401 mvcc.cc:205] Tried to move back new op lower bound from 7299017036995481600 to 7299017036750819328. Current Snapshot: MvccSnapshot[applied={T|T < 7299017036995481600}]
I20260620 20:16:21.319296 22402 mvcc.cc:205] Tried to move back new op lower bound from 7299017036995481600 to 7299017036750819328. Current Snapshot: MvccSnapshot[applied={T|T < 7299017036995481600}]
I20260620 20:16:21.330228 21920 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:59902:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 20:16:21.330518 21920 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:59902:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 20:16:21.333796 21920 catalog_manager.cc:5782] T 00000000000000000000000000000000 P dbea80af3cbf4ca3915c6736ad52aafb: Sending DeleteTablet for 3 replicas of tablet 2fb8429558a7462c8f4de4180481b761
I20260620 20:16:21.334949 22043 tablet_service.cc:1514] Processing DeleteTablet for tablet 2fb8429558a7462c8f4de4180481b761 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:16:21 UTC) from {username='slave'} at 127.0.0.1:32778
I20260620 20:16:21.335206 22174 tablet_service.cc:1514] Processing DeleteTablet for tablet 2fb8429558a7462c8f4de4180481b761 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:16:21 UTC) from {username='slave'} at 127.0.0.1:58372
I20260620 20:16:21.335450 22304 tablet_service.cc:1514] Processing DeleteTablet for tablet 2fb8429558a7462c8f4de4180481b761 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:16:21 UTC) from {username='slave'} at 127.0.0.1:56458
I20260620 20:16:21.336951 17252 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu pid 21981
I20260620 20:16:21.345803 22411 tablet_replica.cc:330] T 2fb8429558a7462c8f4de4180481b761 P 975155771e5147ae9520308c6bb0ca26: stopping tablet replica
I20260620 20:16:21.346020 22410 tablet_replica.cc:330] T 2fb8429558a7462c8f4de4180481b761 P af1184475ce44f2a8ceb6434ac7f13e4: stopping tablet replica
I20260620 20:16:21.346354 22410 raft_consensus.cc:2229] T 2fb8429558a7462c8f4de4180481b761 P af1184475ce44f2a8ceb6434ac7f13e4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 20:16:21.346333 22411 raft_consensus.cc:2229] T 2fb8429558a7462c8f4de4180481b761 P 975155771e5147ae9520308c6bb0ca26 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 20:16:21.346861 22410 raft_consensus.cc:2258] T 2fb8429558a7462c8f4de4180481b761 P af1184475ce44f2a8ceb6434ac7f13e4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:16:21.346861 22411 raft_consensus.cc:2258] T 2fb8429558a7462c8f4de4180481b761 P 975155771e5147ae9520308c6bb0ca26 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:16:21.347666 22412 tablet_replica.cc:330] T 2fb8429558a7462c8f4de4180481b761 P 8485cedc3edc49769c593eacfa36b245: stopping tablet replica
I20260620 20:16:21.348120 22412 raft_consensus.cc:2229] T 2fb8429558a7462c8f4de4180481b761 P 8485cedc3edc49769c593eacfa36b245 [term 1 LEADER]: Raft consensus shutting down.
I20260620 20:16:21.348834 22412 raft_consensus.cc:2258] T 2fb8429558a7462c8f4de4180481b761 P 8485cedc3edc49769c593eacfa36b245 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:16:21.349421 22410 ts_tablet_manager.cc:1905] T 2fb8429558a7462c8f4de4180481b761 P af1184475ce44f2a8ceb6434ac7f13e4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:16:21.351027 22412 ts_tablet_manager.cc:1905] T 2fb8429558a7462c8f4de4180481b761 P 8485cedc3edc49769c593eacfa36b245: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:16:21.354797 22411 ts_tablet_manager.cc:1905] T 2fb8429558a7462c8f4de4180481b761 P 975155771e5147ae9520308c6bb0ca26: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:16:21.355607 22381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:21.338401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32786 (local address 127.16.217.1:39271)
0620 20:16:21.338576 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0620 20:16:21.338582 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:16:21.353119 (+ 14537us) server_negotiation.cc:407] Connection header received
0620 20:16:21.353427 (+ 308us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:16:21.353436 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:16:21.353541 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:16:21.353636 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:16:21.354829 (+ 1193us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:16:21.354835 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:16:21.354839 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:16:21.354857 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:16:21.354892 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:16:21.354909 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:16:21.355009 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0620 20:16:21.355258 (+ 249us) server_negotiation.cc:299] Negotiation successful
0620 20:16:21.355335 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 20:16:21.356871 22103 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:16:21.358711 22412 ts_tablet_manager.cc:1918] T 2fb8429558a7462c8f4de4180481b761 P 8485cedc3edc49769c593eacfa36b245: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 20:16:21.358922 22412 log.cc:1196] T 2fb8429558a7462c8f4de4180481b761 P 8485cedc3edc49769c593eacfa36b245: Deleting WAL directory at /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986536687152-17252-0/minicluster-data/ts-2/wal/wals/2fb8429558a7462c8f4de4180481b761
I20260620 20:16:21.359465 22412 ts_tablet_manager.cc:1939] T 2fb8429558a7462c8f4de4180481b761 P 8485cedc3edc49769c593eacfa36b245: Deleting consensus metadata
I20260620 20:16:21.361311 21909 catalog_manager.cc:4841] TS 8485cedc3edc49769c593eacfa36b245 (127.16.217.3:36545): tablet 2fb8429558a7462c8f4de4180481b761 (table test-table [id=34a734114c744e10b929f4d36d5bab3f]) successfully deleted
I20260620 20:16:21.362602 22411 ts_tablet_manager.cc:1918] T 2fb8429558a7462c8f4de4180481b761 P 975155771e5147ae9520308c6bb0ca26: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 20:16:21.362778 22411 log.cc:1196] T 2fb8429558a7462c8f4de4180481b761 P 975155771e5147ae9520308c6bb0ca26: Deleting WAL directory at /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986536687152-17252-0/minicluster-data/ts-1/wal/wals/2fb8429558a7462c8f4de4180481b761
I20260620 20:16:21.363201 22411 ts_tablet_manager.cc:1939] T 2fb8429558a7462c8f4de4180481b761 P 975155771e5147ae9520308c6bb0ca26: Deleting consensus metadata
I20260620 20:16:21.364630 21909 catalog_manager.cc:4841] TS 975155771e5147ae9520308c6bb0ca26 (127.16.217.2:44253): tablet 2fb8429558a7462c8f4de4180481b761 (table test-table [id=34a734114c744e10b929f4d36d5bab3f]) successfully deleted
I20260620 20:16:21.450552 17252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu with pid 21981
I20260620 20:16:21.473054 17252 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu pid 22112
W20260620 20:16:21.473114 21908 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.16.217.1:39271 (error 108)
I20260620 20:16:21.480705 22385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:21.479891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.1:39271 (local address 127.0.0.1:32798)
0620 20:16:21.480368 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0620 20:16:21.480527 (+ 159us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.217.1:39271: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":340}
I20260620 20:16:21.481304 22386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:21.478839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58374 (local address 127.16.217.2:44253)
0620 20:16:21.479020 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0620 20:16:21.479025 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:16:21.479043 (+ 18us) server_negotiation.cc:407] Connection header received
0620 20:16:21.479106 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:16:21.479113 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:16:21.479207 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:16:21.479339 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:16:21.480255 (+ 916us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:16:21.480261 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:16:21.480267 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:16:21.480287 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:16:21.480325 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:16:21.480341 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:16:21.480462 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0620 20:16:21.480974 (+ 512us) server_negotiation.cc:299] Negotiation successful
0620 20:16:21.481032 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
W20260620 20:16:21.481220 21908 catalog_manager.cc:4584] TS af1184475ce44f2a8ceb6434ac7f13e4 (127.16.217.1:39271): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2fb8429558a7462c8f4de4180481b761: Network error: Client connection negotiation failed: client connection to 127.16.217.1:39271: connect: Connection refused (error 111)
I20260620 20:16:21.482079 22234 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:16:21.534926 22385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:21.534385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.1:39271 (local address 127.0.0.1:32804)
0620 20:16:21.534628 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0620 20:16:21.534735 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.217.1:39271: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
I20260620 20:16:21.536113 22385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:21.535718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.1:39271 (local address 127.0.0.1:32820)
0620 20:16:21.535882 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0620 20:16:21.535979 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.217.1:39271: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260620 20:16:21.571069 17252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu with pid 22112
I20260620 20:16:21.593109 17252 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu pid 22242
I20260620 20:16:21.596132 22384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:21.593901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56490 (local address 127.16.217.3:36545)
0620 20:16:21.594093 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0620 20:16:21.594099 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:16:21.594124 (+ 25us) server_negotiation.cc:407] Connection header received
0620 20:16:21.594327 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:16:21.594334 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:16:21.594410 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:16:21.594523 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:16:21.595441 (+ 918us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:16:21.595447 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:16:21.595452 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:16:21.595473 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:16:21.595512 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:16:21.595527 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:16:21.595663 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0620 20:16:21.595732 (+ 69us) server_negotiation.cc:299] Negotiation successful
0620 20:16:21.595787 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 20:16:21.596807 22385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:21.596352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.1:39271 (local address 127.0.0.1:32822)
0620 20:16:21.596526 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0620 20:16:21.596641 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.217.1:39271: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260620 20:16:21.597115 22364 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:16:21.597965 22385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:21.597524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.1:39271 (local address 127.0.0.1:32836)
0620 20:16:21.597708 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0620 20:16:21.597796 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.217.1:39271: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260620 20:16:21.678917 22385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:21.678286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.1:39271 (local address 127.0.0.1:32840)
0620 20:16:21.678561 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0620 20:16:21.678690 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.217.1:39271: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
I20260620 20:16:21.680128 22385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:21.679662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.1:39271 (local address 127.0.0.1:32844)
0620 20:16:21.679883 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0620 20:16:21.679971 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.217.1:39271: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112}
I20260620 20:16:21.712635 17252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu with pid 22242
I20260620 20:16:21.731024 17252 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu pid 21891
I20260620 20:16:21.732129 21952 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:16:21.835345 17252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu with pid 21891
2026-06-20T20:16:21Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (2829 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260620 20:16:21.876173 17252 mini_oidc.cc:152] Starting JWKS server
I20260620 20:16:21.878677 17252 webserver.cc:460] Webserver started at https://127.0.0.1:43425/ using document root <none> and password file <none>
I20260620 20:16:21.878983 17252 mini_oidc.cc:189] Starting OIDC Discovery server
I20260620 20:16:21.879594 17252 webserver.cc:460] Webserver started at http://127.0.0.1:37945/ using document root <none> and password file <none>
2026-06-20T20:16:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T20:16:21Z Disabled control of system clock
I20260620 20:16:21.907536 17252 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
/tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781986536687152-17252-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781986536687152-17252-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781986536687152-17252-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781986536687152-17252-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.217.62:36905
--webserver_interface=127.16.217.62
--webserver_port=0
--builtin_ntp_servers=127.16.217.20:37159
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.217.62:36905
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43425/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781986536687152-17252-0/testchainca.cert with env {}
W20260620 20:16:22.230681 22424 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:16:22.230973 22424 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:16:22.231057 22424 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:16:22.238232 22424 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:16:22.238346 22424 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:16:22.238384 22424 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:16:22.238431 22424 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:16:22.238476 22424 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 20:16:22.238528 22424 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 20:16:22.238569 22424 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:43425/jwks.json
I20260620 20:16:22.248445 22424 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.217.20:37159
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781986536687152-17252-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781986536687152-17252-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.217.62:36905
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.217.62:36905
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43425/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781986536687152-17252-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.217.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781986536687152-17252-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781986536687152-17252-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:16:22.249722 22424 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:16:22.250947 22424 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:16:22.262601 22429 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:16:22.262562 22430 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:16:22.262625 22432 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:16:22.263401 22424 server_base.cc:1034] running on GCE node
I20260620 20:16:22.264197 22424 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:16:22.265764 22424 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:16:22.266992 22424 hybrid_clock.cc:648] HybridClock initialized: now 1781986582266943 us; error 45 us; skew 500 ppm
I20260620 20:16:22.267344 22424 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 20:16:22.269618 22424 webserver.cc:460] Webserver started at http://127.16.217.62:46537/ using document root <none> and password file <none>
I20260620 20:16:22.270407 22424 fs_manager.cc:359] Metadata directory not provided
I20260620 20:16:22.270556 22424 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:16:22.270907 22424 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:16:22.274008 22424 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781986536687152-17252-0/minicluster-data/master-0/data/instance:
uuid: "15d6135a340f483abca99033129fded5"
format_stamp: "Formatted at 2026-06-20 20:16:22 on dist-test-slave-2hpg"
server_key: "08f43c5c21d44e7f7b1824dfb7259c39"
server_key_iv: "d4d03aefee6ddea08652781f673f1e6b"
server_key_version: "encryptionkey@0"
I20260620 20:16:22.274901 22424 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781986536687152-17252-0/minicluster-data/master-0/wal/instance:
uuid: "15d6135a340f483abca99033129fded5"
format_stamp: "Formatted at 2026-06-20 20:16:22 on dist-test-slave-2hpg"
server_key: "08f43c5c21d44e7f7b1824dfb7259c39"
server_key_iv: "d4d03aefee6ddea08652781f673f1e6b"
server_key_version: "encryptionkey@0"
I20260620 20:16:22.280200 22424 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 20:16:22.283771 22438 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:16:22.285380 22424 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 20:16:22.285599 22424 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781986536687152-17252-0/minicluster-data/master-0/data,/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781986536687152-17252-0/minicluster-data/master-0/wal
uuid: "15d6135a340f483abca99033129fded5"
format_stamp: "Formatted at 2026-06-20 20:16:22 on dist-test-slave-2hpg"
server_key: "08f43c5c21d44e7f7b1824dfb7259c39"
server_key_iv: "d4d03aefee6ddea08652781f673f1e6b"
server_key_version: "encryptionkey@0"
I20260620 20:16:22.285945 22424 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781986536687152-17252-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781986536687152-17252-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781986536687152-17252-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:16:22.321576 22424 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:16:22.322499 22424 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:16:22.322825 22424 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:16:22.341125 22424 rpc_server.cc:307] RPC server started. Bound to: 127.16.217.62:36905
I20260620 20:16:22.341138 22489 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.217.62:36905 every 8 connection(s)
I20260620 20:16:22.342482 22424 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781986536687152-17252-0/minicluster-data/master-0/data/info.pb
I20260620 20:16:22.346267 22490 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:16:22.346297 17252 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu as pid 22424
I20260620 20:16:22.346491 17252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781986536687152-17252-0/minicluster-data/master-0/wal/instance
I20260620 20:16:22.347111 17252 external_mini_cluster.cc:1428] Setting key 22de16760bfe645551320ef59d0fb613
I20260620 20:16:22.360347 22490 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 15d6135a340f483abca99033129fded5: Bootstrap starting.
I20260620 20:16:22.362957 22493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:22.349020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58422 (local address 127.16.217.62:36905)
0620 20:16:22.350137 (+ 1117us) server_negotiation.cc:206] Beginning negotiation
0620 20:16:22.350152 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 20:16:22.350205 (+ 53us) server_negotiation.cc:407] Connection header received
0620 20:16:22.352777 (+ 2572us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:16:22.352804 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:16:22.353439 (+ 635us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:16:22.353923 (+ 484us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:16:22.354414 (+ 491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:22.356244 (+ 1830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:22.357538 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:22.358635 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:22.358814 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:22.360126 (+ 1312us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:16:22.360138 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:16:22.360155 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:16:22.360852 (+ 697us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:16:22.360951 (+ 99us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:16:22.360975 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:16:22.361085 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0620 20:16:22.361288 (+ 203us) server_negotiation.cc:299] Negotiation successful
0620 20:16:22.361614 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":554,"thread_start_us":329,"threads_started":1}
I20260620 20:16:22.364140 22490 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 15d6135a340f483abca99033129fded5: Neither blocks nor log segments found. Creating new log.
I20260620 20:16:22.365264 22490 log.cc:824] T 00000000000000000000000000000000 P 15d6135a340f483abca99033129fded5: Log is configured to *not* fsync() on all Append() calls
I20260620 20:16:22.366760 17252 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 20:16:22.368028 22490 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 15d6135a340f483abca99033129fded5: No bootstrap required, opened a new log
I20260620 20:16:22.375520 22490 raft_consensus.cc:348] T 00000000000000000000000000000000 P 15d6135a340f483abca99033129fded5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15d6135a340f483abca99033129fded5" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 36905 } }
I20260620 20:16:22.375947 22490 raft_consensus.cc:374] T 00000000000000000000000000000000 P 15d6135a340f483abca99033129fded5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:16:22.376068 22490 raft_consensus.cc:729] T 00000000000000000000000000000000 P 15d6135a340f483abca99033129fded5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15d6135a340f483abca99033129fded5, State: Initialized, Role: FOLLOWER
I20260620 20:16:22.376703 22490 consensus_queue.cc:260] T 00000000000000000000000000000000 P 15d6135a340f483abca99033129fded5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15d6135a340f483abca99033129fded5" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 36905 } }
I20260620 20:16:22.376945 22490 raft_consensus.cc:388] T 00000000000000000000000000000000 P 15d6135a340f483abca99033129fded5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 20:16:22.377089 22490 raft_consensus.cc:482] T 00000000000000000000000000000000 P 15d6135a340f483abca99033129fded5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 20:16:22.377261 22490 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 15d6135a340f483abca99033129fded5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:16:22.379475 22490 raft_consensus.cc:504] T 00000000000000000000000000000000 P 15d6135a340f483abca99033129fded5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15d6135a340f483abca99033129fded5" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 36905 } }
I20260620 20:16:22.380018 22490 leader_election.cc:302] T 00000000000000000000000000000000 P 15d6135a340f483abca99033129fded5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 15d6135a340f483abca99033129fded5; no voters:
I20260620 20:16:22.380662 22490 leader_election.cc:288] T 00000000000000000000000000000000 P 15d6135a340f483abca99033129fded5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 20:16:22.380764 22495 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 15d6135a340f483abca99033129fded5 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:16:22.382088 22495 raft_consensus.cc:686] T 00000000000000000000000000000000 P 15d6135a340f483abca99033129fded5 [term 1 LEADER]: Becoming Leader. State: Replica: 15d6135a340f483abca99033129fded5, State: Running, Role: LEADER
I20260620 20:16:22.382609 22495 consensus_queue.cc:237] T 00000000000000000000000000000000 P 15d6135a340f483abca99033129fded5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15d6135a340f483abca99033129fded5" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 36905 } }
I20260620 20:16:22.382993 22490 sys_catalog.cc:564] T 00000000000000000000000000000000 P 15d6135a340f483abca99033129fded5 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:16:22.385438 22497 sys_catalog.cc:455] T 00000000000000000000000000000000 P 15d6135a340f483abca99033129fded5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 15d6135a340f483abca99033129fded5. Latest consensus state: current_term: 1 leader_uuid: "15d6135a340f483abca99033129fded5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15d6135a340f483abca99033129fded5" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 36905 } } }
I20260620 20:16:22.385819 22497 sys_catalog.cc:458] T 00000000000000000000000000000000 P 15d6135a340f483abca99033129fded5 [sys.catalog]: This master's current role is: LEADER
I20260620 20:16:22.386368 22496 sys_catalog.cc:455] T 00000000000000000000000000000000 P 15d6135a340f483abca99033129fded5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "15d6135a340f483abca99033129fded5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15d6135a340f483abca99033129fded5" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 36905 } } }
I20260620 20:16:22.386644 22496 sys_catalog.cc:458] T 00000000000000000000000000000000 P 15d6135a340f483abca99033129fded5 [sys.catalog]: This master's current role is: LEADER
I20260620 20:16:22.388612 22502 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 20:16:22.394232 22502 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 20:16:22.403617 22502 catalog_manager.cc:1347] Generated new cluster ID: dd984d3b272b41aaa2d36871bbf6dea5
I20260620 20:16:22.403775 22502 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 20:16:22.415796 22502 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 20:16:22.417032 22502 catalog_manager.cc:1504] Loading token signing keys...
I20260620 20:16:22.431249 22502 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 15d6135a340f483abca99033129fded5: Generated new TSK 0
I20260620 20:16:22.432121 22502 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260620 20:16:22.465519 22520 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260620 20:16:22.465718 22520 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260620 20:16:22.466837 22493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:22.446009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58426 (local address 127.16.217.62:36905)
0620 20:16:22.446164 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0620 20:16:22.446168 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:16:22.446701 (+ 533us) server_negotiation.cc:407] Connection header received
0620 20:16:22.447048 (+ 347us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:16:22.447052 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:16:22.447131 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:16:22.447272 (+ 141us) server_negotiation.cc:226] Negotiated authn=JWT
0620 20:16:22.447823 (+ 551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:22.448537 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:22.450067 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:22.450501 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:22.450638 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:22.451178 (+ 540us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 20:16:22.466125 (+ 14947us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43425/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0620 20:16:22.466509 (+ 384us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:58426: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43425/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":51}
W20260620 20:16:22.466960 22493 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:58426: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43425/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260620 20:16:22.468569 22519 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:16:22.446019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:36905 (local address 127.0.0.1:58426)
0620 20:16:22.446590 (+ 571us) negotiation.cc:107] Waiting for socket to connect
0620 20:16:22.446607 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 20:16:22.446806 (+ 199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:16:22.447305 (+ 499us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:16:22.447312 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:16:22.447325 (+ 13us) client_negotiation.cc:189] Negotiated authn=JWT
0620 20:16:22.447664 (+ 339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:16:22.447673 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:16:22.448695 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:16:22.448700 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:16:22.449850 (+ 1150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:16:22.449928 (+ 78us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:16:22.450689 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:16:22.450698 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:16:22.450840 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:22.450900 (+ 60us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 20:16:22.466435 (+ 15535us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43425/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0620 20:16:22.466774 (+ 339us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.16.217.62:36905: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43425/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":418,"thread_start_us":294,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.16.217.62:36905: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43425/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260620 20:16:22.471114 17252 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu pid 22424
I20260620 20:16:22.472368 22485 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:16:22.565241 17252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu with pid 22424
2026-06-20T20:16:22Z chronyd exiting
I20260620 20:16:22.615773 17252 test_util.cc:175] -----------------------------------------------
I20260620 20:16:22.615967 17252 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781986536687152-17252-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (744 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260620 20:16:22.620143 17252 mini_oidc.cc:152] Starting JWKS server
I20260620 20:16:22.622218 17252 webserver.cc:460] Webserver started at https://127.0.0.1:36151/ using document root <none> and password file <none>
I20260620 20:16:22.622486 17252 mini_oidc.cc:189] Starting OIDC Discovery server
I20260620 20:16:22.623158 17252 webserver.cc:460] Webserver started at http://127.0.0.1:40491/ using document root <none> and password file <none>
2026-06-20T20:16:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T20:16:22Z Disabled control of system clock
I20260620 20:16:22.650436 17252 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
/tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781986536687152-17252-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781986536687152-17252-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781986536687152-17252-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781986536687152-17252-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.217.62:44047
--webserver_interface=127.16.217.62
--webserver_port=0
--builtin_ntp_servers=127.16.217.20:37637
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.217.62:44047
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36151/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781986536687152-17252-0/testchainca.cert with env {}
W20260620 20:16:22.975098 22527 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:16:22.975437 22527 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:16:22.975589 22527 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:16:22.983219 22527 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:16:22.983346 22527 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:16:22.983412 22527 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:16:22.983495 22527 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:16:22.983559 22527 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 20:16:22.983647 22527 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 20:16:22.983704 22527 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:36151/jwks.json
I20260620 20:16:22.993754 22527 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.217.20:37637
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781986536687152-17252-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781986536687152-17252-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.217.62:44047
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.217.62:44047
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36151/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781986536687152-17252-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.217.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781986536687152-17252-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781986536687152-17252-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:16:22.995015 22527 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:16:22.996297 22527 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:16:23.008399 22532 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:16:23.008322 22533 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:16:23.008373 22535 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:16:23.009526 22527 server_base.cc:1034] running on GCE node
I20260620 20:16:23.010532 22527 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:16:23.012077 22527 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:16:23.013324 22527 hybrid_clock.cc:648] HybridClock initialized: now 1781986583013277 us; error 41 us; skew 500 ppm
I20260620 20:16:23.013703 22527 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 20:16:23.016189 22527 webserver.cc:460] Webserver started at http://127.16.217.62:37479/ using document root <none> and password file <none>
I20260620 20:16:23.016913 22527 fs_manager.cc:359] Metadata directory not provided
I20260620 20:16:23.017051 22527 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:16:23.017373 22527 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:16:23.020246 22527 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781986536687152-17252-0/minicluster-data/master-0/data/instance:
uuid: "41257ee040674bfa8e85d88bee4264bc"
format_stamp: "Formatted at 2026-06-20 20:16:23 on dist-test-slave-2hpg"
server_key: "676d9c582e1fd3d5847e7fa011e4b327"
server_key_iv: "f09622786b6415d9f5ccc92488f25016"
server_key_version: "encryptionkey@0"
I20260620 20:16:23.021019 22527 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781986536687152-17252-0/minicluster-data/master-0/wal/instance:
uuid: "41257ee040674bfa8e85d88bee4264bc"
format_stamp: "Formatted at 2026-06-20 20:16:23 on dist-test-slave-2hpg"
server_key: "676d9c582e1fd3d5847e7fa011e4b327"
server_key_iv: "f09622786b6415d9f5ccc92488f25016"
server_key_version: "encryptionkey@0"
I20260620 20:16:23.026063 22527 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260620 20:16:23.029475 22541 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:16:23.030957 22527 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 20:16:23.031163 22527 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781986536687152-17252-0/minicluster-data/master-0/data,/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781986536687152-17252-0/minicluster-data/master-0/wal
uuid: "41257ee040674bfa8e85d88bee4264bc"
format_stamp: "Formatted at 2026-06-20 20:16:23 on dist-test-slave-2hpg"
server_key: "676d9c582e1fd3d5847e7fa011e4b327"
server_key_iv: "f09622786b6415d9f5ccc92488f25016"
server_key_version: "encryptionkey@0"
I20260620 20:16:23.031404 22527 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781986536687152-17252-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781986536687152-17252-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781986536687152-17252-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:16:23.066022 22527 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:16:23.066913 22527 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:16:23.067232 22527 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:16:23.085513 22527 rpc_server.cc:307] RPC server started. Bound to: 127.16.217.62:44047
I20260620 20:16:23.085536 22592 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.217.62:44047 every 8 connection(s)
I20260620 20:16:23.086951 22527 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781986536687152-17252-0/minicluster-data/master-0/data/info.pb
I20260620 20:16:23.088901 17252 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu as pid 22527
I20260620 20:16:23.089097 17252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781986536687152-17252-0/minicluster-data/master-0/wal/instance
I20260620 20:16:23.089962 17252 external_mini_cluster.cc:1428] Setting key 4d47b6720435f9ffae54558a3bce990d
I20260620 20:16:23.091503 22593 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:16:23.102838 22596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:23.091943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39912 (local address 127.16.217.62:44047)
0620 20:16:23.092879 (+ 936us) server_negotiation.cc:206] Beginning negotiation
0620 20:16:23.092890 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 20:16:23.092931 (+ 41us) server_negotiation.cc:407] Connection header received
0620 20:16:23.094520 (+ 1589us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:16:23.094541 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:16:23.095006 (+ 465us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:16:23.095390 (+ 384us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:16:23.095928 (+ 538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:23.097232 (+ 1304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:23.098699 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:23.099329 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:23.099529 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:23.100641 (+ 1112us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:16:23.100651 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:16:23.100664 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:16:23.101199 (+ 535us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:16:23.101272 (+ 73us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:16:23.101291 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:16:23.101414 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0620 20:16:23.101627 (+ 213us) server_negotiation.cc:299] Negotiation successful
0620 20:16:23.101964 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":526,"thread_start_us":335,"threads_started":1}
I20260620 20:16:23.102929 22593 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 41257ee040674bfa8e85d88bee4264bc: Bootstrap starting.
I20260620 20:16:23.106688 17252 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 20:16:23.106725 22593 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 41257ee040674bfa8e85d88bee4264bc: Neither blocks nor log segments found. Creating new log.
I20260620 20:16:23.107684 22593 log.cc:824] T 00000000000000000000000000000000 P 41257ee040674bfa8e85d88bee4264bc: Log is configured to *not* fsync() on all Append() calls
I20260620 20:16:23.110633 22593 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 41257ee040674bfa8e85d88bee4264bc: No bootstrap required, opened a new log
I20260620 20:16:23.117642 22593 raft_consensus.cc:348] T 00000000000000000000000000000000 P 41257ee040674bfa8e85d88bee4264bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41257ee040674bfa8e85d88bee4264bc" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 44047 } }
I20260620 20:16:23.118062 22593 raft_consensus.cc:374] T 00000000000000000000000000000000 P 41257ee040674bfa8e85d88bee4264bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:16:23.118177 22593 raft_consensus.cc:729] T 00000000000000000000000000000000 P 41257ee040674bfa8e85d88bee4264bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41257ee040674bfa8e85d88bee4264bc, State: Initialized, Role: FOLLOWER
I20260620 20:16:23.118718 22593 consensus_queue.cc:260] T 00000000000000000000000000000000 P 41257ee040674bfa8e85d88bee4264bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41257ee040674bfa8e85d88bee4264bc" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 44047 } }
I20260620 20:16:23.118944 22593 raft_consensus.cc:388] T 00000000000000000000000000000000 P 41257ee040674bfa8e85d88bee4264bc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 20:16:23.119087 22593 raft_consensus.cc:482] T 00000000000000000000000000000000 P 41257ee040674bfa8e85d88bee4264bc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 20:16:23.119252 22593 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 41257ee040674bfa8e85d88bee4264bc [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:16:23.121553 22593 raft_consensus.cc:504] T 00000000000000000000000000000000 P 41257ee040674bfa8e85d88bee4264bc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41257ee040674bfa8e85d88bee4264bc" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 44047 } }
I20260620 20:16:23.122102 22593 leader_election.cc:302] T 00000000000000000000000000000000 P 41257ee040674bfa8e85d88bee4264bc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 41257ee040674bfa8e85d88bee4264bc; no voters:
I20260620 20:16:23.122655 22593 leader_election.cc:288] T 00000000000000000000000000000000 P 41257ee040674bfa8e85d88bee4264bc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 20:16:23.122792 22598 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 41257ee040674bfa8e85d88bee4264bc [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:16:23.123600 22598 raft_consensus.cc:686] T 00000000000000000000000000000000 P 41257ee040674bfa8e85d88bee4264bc [term 1 LEADER]: Becoming Leader. State: Replica: 41257ee040674bfa8e85d88bee4264bc, State: Running, Role: LEADER
I20260620 20:16:23.124125 22598 consensus_queue.cc:237] T 00000000000000000000000000000000 P 41257ee040674bfa8e85d88bee4264bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41257ee040674bfa8e85d88bee4264bc" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 44047 } }
I20260620 20:16:23.124670 22593 sys_catalog.cc:564] T 00000000000000000000000000000000 P 41257ee040674bfa8e85d88bee4264bc [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:16:23.127166 22599 sys_catalog.cc:455] T 00000000000000000000000000000000 P 41257ee040674bfa8e85d88bee4264bc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "41257ee040674bfa8e85d88bee4264bc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41257ee040674bfa8e85d88bee4264bc" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 44047 } } }
I20260620 20:16:23.127203 22600 sys_catalog.cc:455] T 00000000000000000000000000000000 P 41257ee040674bfa8e85d88bee4264bc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 41257ee040674bfa8e85d88bee4264bc. Latest consensus state: current_term: 1 leader_uuid: "41257ee040674bfa8e85d88bee4264bc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41257ee040674bfa8e85d88bee4264bc" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 44047 } } }
I20260620 20:16:23.127606 22599 sys_catalog.cc:458] T 00000000000000000000000000000000 P 41257ee040674bfa8e85d88bee4264bc [sys.catalog]: This master's current role is: LEADER
I20260620 20:16:23.127671 22600 sys_catalog.cc:458] T 00000000000000000000000000000000 P 41257ee040674bfa8e85d88bee4264bc [sys.catalog]: This master's current role is: LEADER
I20260620 20:16:23.129851 22607 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 20:16:23.135193 22607 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 20:16:23.143784 22607 catalog_manager.cc:1347] Generated new cluster ID: 9837abace6514d65b473818f762a1bfe
I20260620 20:16:23.143919 22607 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 20:16:23.152812 22607 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 20:16:23.154111 22607 catalog_manager.cc:1504] Loading token signing keys...
I20260620 20:16:23.161854 22607 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 41257ee040674bfa8e85d88bee4264bc: Generated new TSK 0
I20260620 20:16:23.162683 22607 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260620 20:16:23.208508 22623 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260620 20:16:23.208632 22623 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260620 20:16:23.209952 22596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:23.186924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39922 (local address 127.16.217.62:44047)
0620 20:16:23.187148 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0620 20:16:23.187158 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 20:16:23.187671 (+ 513us) server_negotiation.cc:407] Connection header received
0620 20:16:23.187914 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:16:23.187918 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:16:23.187993 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:16:23.188122 (+ 129us) server_negotiation.cc:226] Negotiated authn=JWT
0620 20:16:23.188724 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:23.189901 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:23.191994 (+ 2093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:23.193262 (+ 1268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:23.193453 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:23.194049 (+ 596us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 20:16:23.209167 (+ 15118us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36151/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0620 20:16:23.209594 (+ 427us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:39922: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36151/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":60}
W20260620 20:16:23.210027 22622 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:16:23.186909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:44047 (local address 127.0.0.1:39922)
0620 20:16:23.187548 (+ 639us) negotiation.cc:107] Waiting for socket to connect
0620 20:16:23.187574 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0620 20:16:23.187738 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:16:23.188218 (+ 480us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:16:23.188224 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:16:23.188245 (+ 21us) client_negotiation.cc:189] Negotiated authn=JWT
0620 20:16:23.188570 (+ 325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:16:23.188583 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:16:23.190089 (+ 1506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:16:23.190097 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:16:23.191828 (+ 1731us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:16:23.191843 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:16:23.193475 (+ 1632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:16:23.193482 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:16:23.193676 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:23.193725 (+ 49us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 20:16:23.209408 (+ 15683us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36151/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0620 20:16:23.209639 (+ 231us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.16.217.62:44047: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36151/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":494,"thread_start_us":329,"threads_started":1}
W20260620 20:16:23.210071 22596 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:39922: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36151/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
I20260620 20:16:23.211293 17252 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu pid 22527
I20260620 20:16:23.212550 22588 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:16:23.304122 17252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu with pid 22527
2026-06-20T20:16:23Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (740 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260620 20:16:23.360607 17252 mini_oidc.cc:152] Starting JWKS server
I20260620 20:16:23.362578 17252 webserver.cc:460] Webserver started at https://127.0.0.1:41209/ using document root <none> and password file <none>
I20260620 20:16:23.362843 17252 mini_oidc.cc:189] Starting OIDC Discovery server
I20260620 20:16:23.363431 17252 webserver.cc:460] Webserver started at http://127.0.0.1:40545/ using document root <none> and password file <none>
2026-06-20T20:16:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T20:16:23Z Disabled control of system clock
I20260620 20:16:23.389381 17252 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
/tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781986536687152-17252-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781986536687152-17252-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781986536687152-17252-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781986536687152-17252-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.217.62:32785
--webserver_interface=127.16.217.62
--webserver_port=0
--builtin_ntp_servers=127.16.217.20:45523
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.217.62:32785
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41209/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260620 20:16:23.706832 22630 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:16:23.707175 22630 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:16:23.707333 22630 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:16:23.714471 22630 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:16:23.714619 22630 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:16:23.714708 22630 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:16:23.714802 22630 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:16:23.714890 22630 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 20:16:23.714982 22630 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 20:16:23.715059 22630 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:41209/jwks.json
I20260620 20:16:23.723999 22630 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.217.20:45523
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781986536687152-17252-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781986536687152-17252-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.217.62:32785
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.217.62:32785
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41209/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781986536687152-17252-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.217.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781986536687152-17252-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:16:23.725400 22630 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:16:23.727209 22630 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:16:23.738996 22638 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:16:23.738945 22635 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:16:23.738971 22636 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:16:23.740195 22630 server_base.cc:1034] running on GCE node
I20260620 20:16:23.740986 22630 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:16:23.742456 22630 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:16:23.743655 22630 hybrid_clock.cc:648] HybridClock initialized: now 1781986583743593 us; error 51 us; skew 500 ppm
I20260620 20:16:23.744041 22630 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 20:16:23.746327 22630 webserver.cc:460] Webserver started at http://127.16.217.62:42157/ using document root <none> and password file <none>
I20260620 20:16:23.747010 22630 fs_manager.cc:359] Metadata directory not provided
I20260620 20:16:23.747118 22630 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:16:23.747458 22630 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:16:23.750674 22630 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781986536687152-17252-0/minicluster-data/master-0/data/instance:
uuid: "25d183461bb94e3cafc54fb6f65b881f"
format_stamp: "Formatted at 2026-06-20 20:16:23 on dist-test-slave-2hpg"
server_key: "db5dc4009a5923a86c2cbf1cfce38e3f"
server_key_iv: "e929cbc15d605dfd38c21264c294d67c"
server_key_version: "encryptionkey@0"
I20260620 20:16:23.751477 22630 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781986536687152-17252-0/minicluster-data/master-0/wal/instance:
uuid: "25d183461bb94e3cafc54fb6f65b881f"
format_stamp: "Formatted at 2026-06-20 20:16:23 on dist-test-slave-2hpg"
server_key: "db5dc4009a5923a86c2cbf1cfce38e3f"
server_key_iv: "e929cbc15d605dfd38c21264c294d67c"
server_key_version: "encryptionkey@0"
I20260620 20:16:23.756583 22630 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 20:16:23.759994 22644 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:16:23.761345 22630 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 20:16:23.761546 22630 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781986536687152-17252-0/minicluster-data/master-0/data,/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781986536687152-17252-0/minicluster-data/master-0/wal
uuid: "25d183461bb94e3cafc54fb6f65b881f"
format_stamp: "Formatted at 2026-06-20 20:16:23 on dist-test-slave-2hpg"
server_key: "db5dc4009a5923a86c2cbf1cfce38e3f"
server_key_iv: "e929cbc15d605dfd38c21264c294d67c"
server_key_version: "encryptionkey@0"
I20260620 20:16:23.761765 22630 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781986536687152-17252-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781986536687152-17252-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781986536687152-17252-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:16:23.799566 22630 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:16:23.800441 22630 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:16:23.800758 22630 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:16:23.818958 22630 rpc_server.cc:307] RPC server started. Bound to: 127.16.217.62:32785
I20260620 20:16:23.818962 22695 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.217.62:32785 every 8 connection(s)
I20260620 20:16:23.820334 22630 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781986536687152-17252-0/minicluster-data/master-0/data/info.pb
I20260620 20:16:23.823935 22696 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:16:23.826527 17252 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu as pid 22630
I20260620 20:16:23.826727 17252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781986536687152-17252-0/minicluster-data/master-0/wal/instance
I20260620 20:16:23.827486 17252 external_mini_cluster.cc:1428] Setting key f177ee2ab073098246069536d6c9a415
I20260620 20:16:23.835680 22696 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 25d183461bb94e3cafc54fb6f65b881f: Bootstrap starting.
I20260620 20:16:23.838828 22696 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 25d183461bb94e3cafc54fb6f65b881f: Neither blocks nor log segments found. Creating new log.
I20260620 20:16:23.839933 22696 log.cc:824] T 00000000000000000000000000000000 P 25d183461bb94e3cafc54fb6f65b881f: Log is configured to *not* fsync() on all Append() calls
I20260620 20:16:23.840842 22699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:23.829420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52822 (local address 127.16.217.62:32785)
0620 20:16:23.830494 (+ 1074us) server_negotiation.cc:206] Beginning negotiation
0620 20:16:23.830505 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 20:16:23.830550 (+ 45us) server_negotiation.cc:407] Connection header received
0620 20:16:23.832152 (+ 1602us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:16:23.832174 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:16:23.832629 (+ 455us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:16:23.833031 (+ 402us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:16:23.833616 (+ 585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:23.834943 (+ 1327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:23.836306 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:23.836981 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:23.837175 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:23.838387 (+ 1212us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:16:23.838395 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:16:23.838408 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:16:23.838935 (+ 527us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:16:23.839059 (+ 124us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:16:23.839084 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:16:23.839223 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0620 20:16:23.839426 (+ 203us) server_negotiation.cc:299] Negotiation successful
0620 20:16:23.839718 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":671,"thread_start_us":421,"threads_started":1}
I20260620 20:16:23.842996 22696 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 25d183461bb94e3cafc54fb6f65b881f: No bootstrap required, opened a new log
I20260620 20:16:23.844614 17252 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 20:16:23.850518 22696 raft_consensus.cc:348] T 00000000000000000000000000000000 P 25d183461bb94e3cafc54fb6f65b881f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25d183461bb94e3cafc54fb6f65b881f" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 32785 } }
I20260620 20:16:23.850975 22696 raft_consensus.cc:374] T 00000000000000000000000000000000 P 25d183461bb94e3cafc54fb6f65b881f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:16:23.851083 22696 raft_consensus.cc:729] T 00000000000000000000000000000000 P 25d183461bb94e3cafc54fb6f65b881f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 25d183461bb94e3cafc54fb6f65b881f, State: Initialized, Role: FOLLOWER
I20260620 20:16:23.851694 22696 consensus_queue.cc:260] T 00000000000000000000000000000000 P 25d183461bb94e3cafc54fb6f65b881f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25d183461bb94e3cafc54fb6f65b881f" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 32785 } }
I20260620 20:16:23.851922 22696 raft_consensus.cc:388] T 00000000000000000000000000000000 P 25d183461bb94e3cafc54fb6f65b881f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 20:16:23.852070 22696 raft_consensus.cc:482] T 00000000000000000000000000000000 P 25d183461bb94e3cafc54fb6f65b881f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 20:16:23.852243 22696 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 25d183461bb94e3cafc54fb6f65b881f [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:16:23.854450 22696 raft_consensus.cc:504] T 00000000000000000000000000000000 P 25d183461bb94e3cafc54fb6f65b881f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25d183461bb94e3cafc54fb6f65b881f" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 32785 } }
I20260620 20:16:23.854938 22696 leader_election.cc:302] T 00000000000000000000000000000000 P 25d183461bb94e3cafc54fb6f65b881f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 25d183461bb94e3cafc54fb6f65b881f; no voters:
I20260620 20:16:23.855557 22696 leader_election.cc:288] T 00000000000000000000000000000000 P 25d183461bb94e3cafc54fb6f65b881f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 20:16:23.855680 22701 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 25d183461bb94e3cafc54fb6f65b881f [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:16:23.857061 22701 raft_consensus.cc:686] T 00000000000000000000000000000000 P 25d183461bb94e3cafc54fb6f65b881f [term 1 LEADER]: Becoming Leader. State: Replica: 25d183461bb94e3cafc54fb6f65b881f, State: Running, Role: LEADER
I20260620 20:16:23.857527 22696 sys_catalog.cc:564] T 00000000000000000000000000000000 P 25d183461bb94e3cafc54fb6f65b881f [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:16:23.857554 22701 consensus_queue.cc:237] T 00000000000000000000000000000000 P 25d183461bb94e3cafc54fb6f65b881f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25d183461bb94e3cafc54fb6f65b881f" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 32785 } }
I20260620 20:16:23.860654 22703 sys_catalog.cc:455] T 00000000000000000000000000000000 P 25d183461bb94e3cafc54fb6f65b881f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 25d183461bb94e3cafc54fb6f65b881f. Latest consensus state: current_term: 1 leader_uuid: "25d183461bb94e3cafc54fb6f65b881f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25d183461bb94e3cafc54fb6f65b881f" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 32785 } } }
I20260620 20:16:23.860673 22702 sys_catalog.cc:455] T 00000000000000000000000000000000 P 25d183461bb94e3cafc54fb6f65b881f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "25d183461bb94e3cafc54fb6f65b881f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25d183461bb94e3cafc54fb6f65b881f" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 32785 } } }
I20260620 20:16:23.861121 22703 sys_catalog.cc:458] T 00000000000000000000000000000000 P 25d183461bb94e3cafc54fb6f65b881f [sys.catalog]: This master's current role is: LEADER
I20260620 20:16:23.861122 22702 sys_catalog.cc:458] T 00000000000000000000000000000000 P 25d183461bb94e3cafc54fb6f65b881f [sys.catalog]: This master's current role is: LEADER
I20260620 20:16:23.864903 22711 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 20:16:23.869019 22711 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 20:16:23.876981 22711 catalog_manager.cc:1347] Generated new cluster ID: f917251e8e24422490efc727c7f49983
I20260620 20:16:23.877089 22711 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 20:16:23.905437 22711 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 20:16:23.907153 22711 catalog_manager.cc:1504] Loading token signing keys...
I20260620 20:16:23.914884 22711 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 25d183461bb94e3cafc54fb6f65b881f: Generated new TSK 0
I20260620 20:16:23.915798 22711 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 20:16:23.935734 22699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:23.927626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52828 (local address 127.16.217.62:32785)
0620 20:16:23.927904 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0620 20:16:23.927911 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:16:23.928689 (+ 778us) server_negotiation.cc:407] Connection header received
0620 20:16:23.929148 (+ 459us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:16:23.929154 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:16:23.929288 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:16:23.929460 (+ 172us) server_negotiation.cc:226] Negotiated authn=JWT
0620 20:16:23.929947 (+ 487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:23.930708 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:23.932074 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:23.932537 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:23.932678 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:23.933202 (+ 524us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 20:16:23.935081 (+ 1879us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41209/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0620 20:16:23.935404 (+ 323us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:52828: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41209/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":72}
W20260620 20:16:23.935989 22699 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:52828: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41209/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260620 20:16:23.936172 22725 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:16:23.927402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:32785 (local address 127.0.0.1:52828)
0620 20:16:23.928533 (+ 1131us) negotiation.cc:107] Waiting for socket to connect
0620 20:16:23.928574 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 20:16:23.928856 (+ 282us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:16:23.929482 (+ 626us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:16:23.929487 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:16:23.929499 (+ 12us) client_negotiation.cc:189] Negotiated authn=JWT
0620 20:16:23.929765 (+ 266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:16:23.929780 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:16:23.930860 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:16:23.930868 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:16:23.931927 (+ 1059us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:16:23.931937 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:16:23.932740 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:16:23.932749 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:16:23.932898 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:23.932932 (+ 34us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 20:16:23.935262 (+ 2330us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41209/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0620 20:16:23.935514 (+ 252us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.16.217.62:32785: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41209/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":987,"thread_start_us":813,"threads_started":1}
I20260620 20:16:23.938114 17252 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu pid 22630
I20260620 20:16:23.939208 22691 generic_service.cc:196] Checking for leaks (request via RPC)
E20260620 20:16:23.948575 22726 webserver.cc:533] Webserver: sslize failed:
E20260620 20:16:23.948674 22726 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260620 20:16:24.030308 17252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu with pid 22630
2026-06-20T20:16:24Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (724 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-06-20T20:16:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T20:16:24Z Disabled control of system clock
I20260620 20:16:24.111765 17252 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
/tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781986536687152-17252-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781986536687152-17252-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781986536687152-17252-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781986536687152-17252-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.217.62:42751
--webserver_interface=127.16.217.62
--webserver_port=0
--builtin_ntp_servers=127.16.217.20:40725
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.217.62:42751
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781986536687152-17252-0/testchainca.cert with env {}
W20260620 20:16:24.429409 22731 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:16:24.429697 22731 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:16:24.429791 22731 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:16:24.436925 22731 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:16:24.437027 22731 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:16:24.437065 22731 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:16:24.437124 22731 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:16:24.437168 22731 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 20:16:24.437222 22731 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 20:16:24.437260 22731 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260620 20:16:24.447427 22731 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.217.20:40725
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781986536687152-17252-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781986536687152-17252-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.217.62:42751
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.217.62:42751
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781986536687152-17252-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.217.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781986536687152-17252-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781986536687152-17252-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:16:24.448737 22731 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:16:24.450021 22731 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:16:24.462576 22739 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:16:24.462659 22736 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:16:24.462662 22737 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:16:24.463935 22731 server_base.cc:1034] running on GCE node
I20260620 20:16:24.464797 22731 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:16:24.466290 22731 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:16:24.467495 22731 hybrid_clock.cc:648] HybridClock initialized: now 1781986584467451 us; error 50 us; skew 500 ppm
I20260620 20:16:24.467833 22731 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 20:16:24.470247 22731 webserver.cc:460] Webserver started at http://127.16.217.62:44137/ using document root <none> and password file <none>
I20260620 20:16:24.470932 22731 fs_manager.cc:359] Metadata directory not provided
I20260620 20:16:24.471021 22731 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:16:24.471279 22731 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:16:24.474156 22731 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781986536687152-17252-0/minicluster-data/master-0/data/instance:
uuid: "1b924bf1d1864d93bc6a3c3b82085005"
format_stamp: "Formatted at 2026-06-20 20:16:24 on dist-test-slave-2hpg"
server_key: "993157f890dcd215059b273286a82a15"
server_key_iv: "e0c2465d2b6da8cd7b634dc697ff26b4"
server_key_version: "encryptionkey@0"
I20260620 20:16:24.474902 22731 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781986536687152-17252-0/minicluster-data/master-0/wal/instance:
uuid: "1b924bf1d1864d93bc6a3c3b82085005"
format_stamp: "Formatted at 2026-06-20 20:16:24 on dist-test-slave-2hpg"
server_key: "993157f890dcd215059b273286a82a15"
server_key_iv: "e0c2465d2b6da8cd7b634dc697ff26b4"
server_key_version: "encryptionkey@0"
I20260620 20:16:24.479931 22731 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260620 20:16:24.483229 22745 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:16:24.484663 22731 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260620 20:16:24.484879 22731 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781986536687152-17252-0/minicluster-data/master-0/data,/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781986536687152-17252-0/minicluster-data/master-0/wal
uuid: "1b924bf1d1864d93bc6a3c3b82085005"
format_stamp: "Formatted at 2026-06-20 20:16:24 on dist-test-slave-2hpg"
server_key: "993157f890dcd215059b273286a82a15"
server_key_iv: "e0c2465d2b6da8cd7b634dc697ff26b4"
server_key_version: "encryptionkey@0"
I20260620 20:16:24.485118 22731 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781986536687152-17252-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781986536687152-17252-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781986536687152-17252-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:16:24.522773 22731 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:16:24.523708 22731 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:16:24.524006 22731 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:16:24.542212 22731 rpc_server.cc:307] RPC server started. Bound to: 127.16.217.62:42751
I20260620 20:16:24.542233 22796 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.217.62:42751 every 8 connection(s)
I20260620 20:16:24.543558 22731 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781986536687152-17252-0/minicluster-data/master-0/data/info.pb
I20260620 20:16:24.547165 22797 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:16:24.550282 17252 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu as pid 22731
I20260620 20:16:24.550477 17252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781986536687152-17252-0/minicluster-data/master-0/wal/instance
I20260620 20:16:24.551123 17252 external_mini_cluster.cc:1428] Setting key b31b7dd2baf6f83f2fb10d18ac82003f
I20260620 20:16:24.559558 22797 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1b924bf1d1864d93bc6a3c3b82085005: Bootstrap starting.
I20260620 20:16:24.563187 22797 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1b924bf1d1864d93bc6a3c3b82085005: Neither blocks nor log segments found. Creating new log.
I20260620 20:16:24.564316 22797 log.cc:824] T 00000000000000000000000000000000 P 1b924bf1d1864d93bc6a3c3b82085005: Log is configured to *not* fsync() on all Append() calls
I20260620 20:16:24.564841 22800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:24.553081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44078 (local address 127.16.217.62:42751)
0620 20:16:24.554125 (+ 1044us) server_negotiation.cc:206] Beginning negotiation
0620 20:16:24.554136 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 20:16:24.554178 (+ 42us) server_negotiation.cc:407] Connection header received
0620 20:16:24.555720 (+ 1542us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:16:24.555743 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:16:24.556233 (+ 490us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:16:24.556573 (+ 340us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:16:24.557206 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:24.558638 (+ 1432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:24.560365 (+ 1727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:24.561027 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:24.561158 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:24.562504 (+ 1346us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:16:24.562511 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:16:24.562524 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:16:24.563075 (+ 551us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:16:24.563160 (+ 85us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:16:24.563183 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:16:24.563304 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0620 20:16:24.563514 (+ 210us) server_negotiation.cc:299] Negotiation successful
0620 20:16:24.563843 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":568,"thread_start_us":347,"threads_started":1}
I20260620 20:16:24.567682 22797 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1b924bf1d1864d93bc6a3c3b82085005: No bootstrap required, opened a new log
I20260620 20:16:24.569008 17252 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 20:16:24.576495 22797 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1b924bf1d1864d93bc6a3c3b82085005 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b924bf1d1864d93bc6a3c3b82085005" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 42751 } }
I20260620 20:16:24.576831 22797 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1b924bf1d1864d93bc6a3c3b82085005 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:16:24.576906 22797 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1b924bf1d1864d93bc6a3c3b82085005 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b924bf1d1864d93bc6a3c3b82085005, State: Initialized, Role: FOLLOWER
I20260620 20:16:24.577391 22797 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1b924bf1d1864d93bc6a3c3b82085005 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b924bf1d1864d93bc6a3c3b82085005" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 42751 } }
I20260620 20:16:24.577575 22797 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1b924bf1d1864d93bc6a3c3b82085005 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 20:16:24.577667 22797 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1b924bf1d1864d93bc6a3c3b82085005 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 20:16:24.577787 22797 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1b924bf1d1864d93bc6a3c3b82085005 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:16:24.580040 22797 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1b924bf1d1864d93bc6a3c3b82085005 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b924bf1d1864d93bc6a3c3b82085005" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 42751 } }
I20260620 20:16:24.580618 22797 leader_election.cc:302] T 00000000000000000000000000000000 P 1b924bf1d1864d93bc6a3c3b82085005 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1b924bf1d1864d93bc6a3c3b82085005; no voters:
I20260620 20:16:24.581220 22797 leader_election.cc:288] T 00000000000000000000000000000000 P 1b924bf1d1864d93bc6a3c3b82085005 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 20:16:24.581358 22802 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1b924bf1d1864d93bc6a3c3b82085005 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:16:24.582135 22802 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1b924bf1d1864d93bc6a3c3b82085005 [term 1 LEADER]: Becoming Leader. State: Replica: 1b924bf1d1864d93bc6a3c3b82085005, State: Running, Role: LEADER
I20260620 20:16:24.582679 22802 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1b924bf1d1864d93bc6a3c3b82085005 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b924bf1d1864d93bc6a3c3b82085005" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 42751 } }
I20260620 20:16:24.583395 22797 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1b924bf1d1864d93bc6a3c3b82085005 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:16:24.585520 22804 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1b924bf1d1864d93bc6a3c3b82085005 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1b924bf1d1864d93bc6a3c3b82085005. Latest consensus state: current_term: 1 leader_uuid: "1b924bf1d1864d93bc6a3c3b82085005" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b924bf1d1864d93bc6a3c3b82085005" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 42751 } } }
I20260620 20:16:24.585530 22803 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1b924bf1d1864d93bc6a3c3b82085005 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1b924bf1d1864d93bc6a3c3b82085005" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b924bf1d1864d93bc6a3c3b82085005" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 42751 } } }
I20260620 20:16:24.585932 22804 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1b924bf1d1864d93bc6a3c3b82085005 [sys.catalog]: This master's current role is: LEADER
I20260620 20:16:24.586079 22803 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1b924bf1d1864d93bc6a3c3b82085005 [sys.catalog]: This master's current role is: LEADER
I20260620 20:16:24.587692 22807 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 20:16:24.592869 22807 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 20:16:24.601854 22807 catalog_manager.cc:1347] Generated new cluster ID: 04b944b1cd984435b6a9fd600dd9be7a
I20260620 20:16:24.602023 22807 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 20:16:24.614409 22807 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 20:16:24.615898 22807 catalog_manager.cc:1504] Loading token signing keys...
I20260620 20:16:24.621697 22807 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1b924bf1d1864d93bc6a3c3b82085005: Generated new TSK 0
I20260620 20:16:24.622573 22807 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 20:16:24.723009 22800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:24.644490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44084 (local address 127.16.217.62:42751)
0620 20:16:24.644668 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0620 20:16:24.644675 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:16:24.645181 (+ 506us) server_negotiation.cc:407] Connection header received
0620 20:16:24.645382 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:16:24.645389 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:16:24.645482 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:16:24.645638 (+ 156us) server_negotiation.cc:226] Negotiated authn=JWT
0620 20:16:24.646403 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:24.647686 (+ 1283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:24.649718 (+ 2032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:24.650496 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:24.650630 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:24.651202 (+ 572us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 20:16:24.722408 (+ 71206us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0620 20:16:24.722759 (+ 351us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:44084: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":41}
W20260620 20:16:24.723182 22800 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:44084: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260620 20:16:24.723474 22826 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:16:24.644398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:42751 (local address 127.0.0.1:44084)
0620 20:16:24.645072 (+ 674us) negotiation.cc:107] Waiting for socket to connect
0620 20:16:24.645089 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 20:16:24.645209 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:16:24.645832 (+ 623us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:16:24.645838 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:16:24.645851 (+ 13us) client_negotiation.cc:189] Negotiated authn=JWT
0620 20:16:24.646196 (+ 345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:16:24.646210 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:16:24.647857 (+ 1647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:16:24.647865 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:16:24.649567 (+ 1702us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:16:24.649583 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:16:24.650684 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:16:24.650692 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:16:24.650878 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:24.650923 (+ 45us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 20:16:24.722734 (+ 71811us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0620 20:16:24.723047 (+ 313us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.16.217.62:42751: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":524,"thread_start_us":369,"threads_started":1}
I20260620 20:16:24.724555 17252 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu pid 22731
I20260620 20:16:24.725757 22792 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:16:24.820037 17252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu with pid 22731
2026-06-20T20:16:24Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (769 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (409 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (404 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986536687152-17252-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 20:16:25 dist-test-slave-2hpg krb5kdc[22841](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 20:16:25 dist-test-slave-2hpg krb5kdc[22841](info): set up 2 sockets
Jun 20 20:16:25 dist-test-slave-2hpg krb5kdc[22841](info): commencing operation
krb5kdc: starting...
W20260620 20:16:26.710837 17252 mini_kdc.cc:121] Time spent starting KDC: real 1.040s user 0.002s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986536687152-17252-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986536687152-17252-0/krb5kdc/test-user.keytab.
Jun 20 20:16:26 dist-test-slave-2hpg krb5kdc[22841](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986586, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T20:16:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T20:16:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.217.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.217.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.217.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.62.keytab.
Entry for principal kudu/127.16.217.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.62.keytab.
I20260620 20:16:26.913993 17252 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
/tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986536687152-17252-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986536687152-17252-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986536687152-17252-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986536687152-17252-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.217.62:44239
--webserver_interface=127.16.217.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.62.keytab
--principal=kudu/127.16.217.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.217.20:42447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.217.62:44239
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986536687152-17252-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986536687152-17252-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986536687152-17252-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:16:27.231376 22854 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:16:27.231712 22854 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:16:27.231870 22854 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:16:27.239107 22854 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:16:27.239254 22854 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:16:27.239364 22854 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:16:27.239459 22854 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:16:27.239539 22854 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:16:27.249516 22854 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.217.20:42447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986536687152-17252-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986536687152-17252-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.217.62:44239
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.62.keytab
--principal=kudu/127.16.217.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.217.62:44239
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986536687152-17252-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.217.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986536687152-17252-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:16:27.251035 22854 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:16:27.252375 22854 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:16:27.264201 22862 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:16:27.264283 22860 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:16:27.264425 22859 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:16:27.265409 22854 server_base.cc:1034] running on GCE node
I20260620 20:16:27.266286 22854 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:16:27.267668 22854 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:16:27.268925 22854 hybrid_clock.cc:648] HybridClock initialized: now 1781986587268878 us; error 35 us; skew 500 ppm
I20260620 20:16:27.269318 22854 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:16:27 dist-test-slave-2hpg krb5kdc[22841](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986587, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.217.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:16:27.272737 22854 init.cc:377] Logged in from keytab as kudu/127.16.217.62@KRBTEST.COM (short username kudu)
I20260620 20:16:27.274416 22854 webserver.cc:460] Webserver started at http://127.16.217.62:45329/ using document root <none> and password file <none>
I20260620 20:16:27.275120 22854 fs_manager.cc:359] Metadata directory not provided
I20260620 20:16:27.275285 22854 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:16:27.275631 22854 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:16:27.278766 22854 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986536687152-17252-0/minicluster-data/master-0/data/instance:
uuid: "127150fca1774699b7e8bc856e169fbe"
format_stamp: "Formatted at 2026-06-20 20:16:27 on dist-test-slave-2hpg"
server_key: "72de2e607cd241f78cd0b1cfb7a7bcce"
server_key_iv: "e7bfa86c882a787e76827a2fb2e65c15"
server_key_version: "encryptionkey@0"
I20260620 20:16:27.279575 22854 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986536687152-17252-0/minicluster-data/master-0/wal/instance:
uuid: "127150fca1774699b7e8bc856e169fbe"
format_stamp: "Formatted at 2026-06-20 20:16:27 on dist-test-slave-2hpg"
server_key: "72de2e607cd241f78cd0b1cfb7a7bcce"
server_key_iv: "e7bfa86c882a787e76827a2fb2e65c15"
server_key_version: "encryptionkey@0"
I20260620 20:16:27.284893 22854 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.005s
I20260620 20:16:27.288276 22869 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:16:27.289584 22854 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 20:16:27.289776 22854 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986536687152-17252-0/minicluster-data/master-0/data,/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986536687152-17252-0/minicluster-data/master-0/wal
uuid: "127150fca1774699b7e8bc856e169fbe"
format_stamp: "Formatted at 2026-06-20 20:16:27 on dist-test-slave-2hpg"
server_key: "72de2e607cd241f78cd0b1cfb7a7bcce"
server_key_iv: "e7bfa86c882a787e76827a2fb2e65c15"
server_key_version: "encryptionkey@0"
I20260620 20:16:27.290016 22854 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986536687152-17252-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986536687152-17252-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986536687152-17252-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:16:27.309702 22854 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:16:27.314067 22854 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:16:27.314353 22854 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:16:27.332825 22854 rpc_server.cc:307] RPC server started. Bound to: 127.16.217.62:44239
I20260620 20:16:27.332903 22920 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.217.62:44239 every 8 connection(s)
I20260620 20:16:27.334275 22854 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986536687152-17252-0/minicluster-data/master-0/data/info.pb
I20260620 20:16:27.337911 22921 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:16:27.342667 17252 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu as pid 22854
I20260620 20:16:27.342888 17252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986536687152-17252-0/minicluster-data/master-0/wal/instance
I20260620 20:16:27.343658 17252 external_mini_cluster.cc:1428] Setting key 58f4044a56f86bdda6fa9be59d8d96e4
I20260620 20:16:27.349094 22921 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 127150fca1774699b7e8bc856e169fbe: Bootstrap starting.
I20260620 20:16:27.352069 22921 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 127150fca1774699b7e8bc856e169fbe: Neither blocks nor log segments found. Creating new log.
I20260620 20:16:27.353071 22921 log.cc:824] T 00000000000000000000000000000000 P 127150fca1774699b7e8bc856e169fbe: Log is configured to *not* fsync() on all Append() calls
I20260620 20:16:27.356091 22921 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 127150fca1774699b7e8bc856e169fbe: No bootstrap required, opened a new log
Jun 20 20:16:27 dist-test-slave-2hpg krb5kdc[22841](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986586, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.217.62@KRBTEST.COM
I20260620 20:16:27.365188 22921 raft_consensus.cc:348] T 00000000000000000000000000000000 P 127150fca1774699b7e8bc856e169fbe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "127150fca1774699b7e8bc856e169fbe" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 44239 } }
I20260620 20:16:27.365847 22921 raft_consensus.cc:374] T 00000000000000000000000000000000 P 127150fca1774699b7e8bc856e169fbe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:16:27.366037 22921 raft_consensus.cc:729] T 00000000000000000000000000000000 P 127150fca1774699b7e8bc856e169fbe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 127150fca1774699b7e8bc856e169fbe, State: Initialized, Role: FOLLOWER
I20260620 20:16:27.366811 22924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:27.345750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59656 (local address 127.16.217.62:44239)
0620 20:16:27.346569 (+ 819us) server_negotiation.cc:206] Beginning negotiation
0620 20:16:27.346581 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 20:16:27.346626 (+ 45us) server_negotiation.cc:407] Connection header received
0620 20:16:27.348481 (+ 1855us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:16:27.348504 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:16:27.348900 (+ 396us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:16:27.349297 (+ 397us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:16:27.350885 (+ 1588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:27.352276 (+ 1391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:27.353856 (+ 1580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:27.354736 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:27.354913 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:27.359195 (+ 4282us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:16:27.359203 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:16:27.359216 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:16:27.359265 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:16:27.362448 (+ 3183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:16:27.363250 (+ 802us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:16:27.363284 (+ 34us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:16:27.363294 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:16:27.363464 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:16:27.363868 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:16:27.363871 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:16:27.363873 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:16:27.364428 (+ 555us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:16:27.364633 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0620 20:16:27.365069 (+ 436us) server_negotiation.cc:299] Negotiation successful
0620 20:16:27.365407 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":600,"thread_start_us":356,"threads_started":1}
I20260620 20:16:27.366670 22921 consensus_queue.cc:260] T 00000000000000000000000000000000 P 127150fca1774699b7e8bc856e169fbe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "127150fca1774699b7e8bc856e169fbe" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 44239 } }
I20260620 20:16:27.367002 22921 raft_consensus.cc:388] T 00000000000000000000000000000000 P 127150fca1774699b7e8bc856e169fbe [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 20:16:27.367185 22921 raft_consensus.cc:482] T 00000000000000000000000000000000 P 127150fca1774699b7e8bc856e169fbe [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 20:16:27.367399 22921 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 127150fca1774699b7e8bc856e169fbe [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:16:27.370311 22921 raft_consensus.cc:504] T 00000000000000000000000000000000 P 127150fca1774699b7e8bc856e169fbe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "127150fca1774699b7e8bc856e169fbe" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 44239 } }
I20260620 20:16:27.370810 22921 leader_election.cc:302] T 00000000000000000000000000000000 P 127150fca1774699b7e8bc856e169fbe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 127150fca1774699b7e8bc856e169fbe; no voters:
I20260620 20:16:27.371349 22921 leader_election.cc:288] T 00000000000000000000000000000000 P 127150fca1774699b7e8bc856e169fbe [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 20:16:27.371481 22926 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 127150fca1774699b7e8bc856e169fbe [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:16:27.372247 22926 raft_consensus.cc:686] T 00000000000000000000000000000000 P 127150fca1774699b7e8bc856e169fbe [term 1 LEADER]: Becoming Leader. State: Replica: 127150fca1774699b7e8bc856e169fbe, State: Running, Role: LEADER
I20260620 20:16:27.372805 22926 consensus_queue.cc:237] T 00000000000000000000000000000000 P 127150fca1774699b7e8bc856e169fbe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "127150fca1774699b7e8bc856e169fbe" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 44239 } }
I20260620 20:16:27.373502 22921 sys_catalog.cc:564] T 00000000000000000000000000000000 P 127150fca1774699b7e8bc856e169fbe [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:16:27.375365 22927 sys_catalog.cc:455] T 00000000000000000000000000000000 P 127150fca1774699b7e8bc856e169fbe [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "127150fca1774699b7e8bc856e169fbe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "127150fca1774699b7e8bc856e169fbe" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 44239 } } }
I20260620 20:16:27.375702 22927 sys_catalog.cc:458] T 00000000000000000000000000000000 P 127150fca1774699b7e8bc856e169fbe [sys.catalog]: This master's current role is: LEADER
I20260620 20:16:27.377740 22928 sys_catalog.cc:455] T 00000000000000000000000000000000 P 127150fca1774699b7e8bc856e169fbe [sys.catalog]: SysCatalogTable state changed. Reason: New leader 127150fca1774699b7e8bc856e169fbe. Latest consensus state: current_term: 1 leader_uuid: "127150fca1774699b7e8bc856e169fbe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "127150fca1774699b7e8bc856e169fbe" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 44239 } } }
I20260620 20:16:27.378211 22928 sys_catalog.cc:458] T 00000000000000000000000000000000 P 127150fca1774699b7e8bc856e169fbe [sys.catalog]: This master's current role is: LEADER
I20260620 20:16:27.378528 22932 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 20:16:27.384963 22932 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 20:16:27.393672 22932 catalog_manager.cc:1347] Generated new cluster ID: 9bc91c658e8e46e390cd61dff5f8e8be
I20260620 20:16:27.393832 22932 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 20:16:27.403534 22932 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 20:16:27.404897 22932 catalog_manager.cc:1504] Loading token signing keys...
I20260620 20:16:27.412052 22932 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 127150fca1774699b7e8bc856e169fbe: Generated new TSK 0
I20260620 20:16:27.412905 22932 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.217.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.217.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.217.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.1.keytab.
Entry for principal kudu/127.16.217.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.1.keytab.
I20260620 20:16:27.466048 17252 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
/tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986536687152-17252-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986536687152-17252-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986536687152-17252-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986536687152-17252-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.217.1:0
--local_ip_for_outbound_sockets=127.16.217.1
--webserver_interface=127.16.217.1
--webserver_port=0
--tserver_master_addrs=127.16.217.62:44239
--keytab_file=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.1.keytab
--principal=kudu/127.16.217.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.217.20:42447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986536687152-17252-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986536687152-17252-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986536687152-17252-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:16:27.787868 22947 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:16:27.788172 22947 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:16:27.788364 22947 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:16:27.795771 22947 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:16:27.795917 22947 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:16:27.796095 22947 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.217.1
I20260620 20:16:27.805701 22947 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.217.20:42447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986536687152-17252-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986536687152-17252-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.1.keytab
--principal=kudu/127.16.217.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.217.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986536687152-17252-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.217.1
--webserver_port=0
--tserver_master_addrs=127.16.217.62:44239
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.217.1
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986536687152-17252-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:16:27.807091 22947 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:16:27.808521 22947 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:16:27.821389 22952 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:16:27.821352 22955 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:16:27.821425 22953 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:16:27.822222 22947 server_base.cc:1034] running on GCE node
I20260620 20:16:27.822849 22947 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:16:27.823822 22947 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:16:27.825067 22947 hybrid_clock.cc:648] HybridClock initialized: now 1781986587825017 us; error 41 us; skew 500 ppm
I20260620 20:16:27.825421 22947 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:16:27 dist-test-slave-2hpg krb5kdc[22841](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986587, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.217.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:16:27.828665 22947 init.cc:377] Logged in from keytab as kudu/127.16.217.1@KRBTEST.COM (short username kudu)
I20260620 20:16:27.830173 22947 webserver.cc:460] Webserver started at http://127.16.217.1:34389/ using document root <none> and password file <none>
I20260620 20:16:27.830843 22947 fs_manager.cc:359] Metadata directory not provided
I20260620 20:16:27.830984 22947 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:16:27.831291 22947 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:16:27.834167 22947 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986536687152-17252-0/minicluster-data/ts-0/data/instance:
uuid: "4fd4d1a935514a81bb53d328f066b287"
format_stamp: "Formatted at 2026-06-20 20:16:27 on dist-test-slave-2hpg"
server_key: "14a33ed7500df877eecf21900a41a254"
server_key_iv: "03ced54ba4a4b2aa7e14bf0b68daea40"
server_key_version: "encryptionkey@0"
I20260620 20:16:27.834935 22947 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986536687152-17252-0/minicluster-data/ts-0/wal/instance:
uuid: "4fd4d1a935514a81bb53d328f066b287"
format_stamp: "Formatted at 2026-06-20 20:16:27 on dist-test-slave-2hpg"
server_key: "14a33ed7500df877eecf21900a41a254"
server_key_iv: "03ced54ba4a4b2aa7e14bf0b68daea40"
server_key_version: "encryptionkey@0"
I20260620 20:16:27.839857 22947 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 20:16:27.843186 22962 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:16:27.844421 22947 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 20:16:27.844595 22947 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986536687152-17252-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986536687152-17252-0/minicluster-data/ts-0/wal
uuid: "4fd4d1a935514a81bb53d328f066b287"
format_stamp: "Formatted at 2026-06-20 20:16:27 on dist-test-slave-2hpg"
server_key: "14a33ed7500df877eecf21900a41a254"
server_key_iv: "03ced54ba4a4b2aa7e14bf0b68daea40"
server_key_version: "encryptionkey@0"
I20260620 20:16:27.844787 22947 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986536687152-17252-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986536687152-17252-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986536687152-17252-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:16:27.867273 22947 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:16:27.871788 22947 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:16:27.872030 22947 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:16:27.872905 22947 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:16:27.874323 22947 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:16:27.874435 22947 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:16:27.874542 22947 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:16:27.874622 22947 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:16:27.907557 22947 rpc_server.cc:307] RPC server started. Bound to: 127.16.217.1:35279
I20260620 20:16:27.907615 23074 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.217.1:35279 every 8 connection(s)
I20260620 20:16:27.908939 22947 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986536687152-17252-0/minicluster-data/ts-0/data/info.pb
I20260620 20:16:27.917939 17252 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu as pid 22947
I20260620 20:16:27.918133 17252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986536687152-17252-0/minicluster-data/ts-0/wal/instance
I20260620 20:16:27.918767 17252 external_mini_cluster.cc:1428] Setting key 3e8914fd7a27d25dc4e50bba206b887e
Jun 20 20:16:27 dist-test-slave-2hpg krb5kdc[22841](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986587, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.217.1@KRBTEST.COM for kudu/127.16.217.62@KRBTEST.COM
I20260620 20:16:27.930401 23077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:27.911387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.217.1:55091 (local address 127.16.217.62:44239)
0620 20:16:27.911864 (+ 477us) server_negotiation.cc:206] Beginning negotiation
0620 20:16:27.911868 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:16:27.912822 (+ 954us) server_negotiation.cc:407] Connection header received
0620 20:16:27.914729 (+ 1907us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:16:27.914734 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:16:27.914832 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:16:27.914947 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:16:27.916397 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:27.917098 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:27.918798 (+ 1700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:27.919272 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:27.919409 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:27.924045 (+ 4636us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:16:27.924054 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:16:27.924063 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:16:27.924107 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:16:27.927474 (+ 3367us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:16:27.928179 (+ 705us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:16:27.928190 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:16:27.928194 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:16:27.928299 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:16:27.928720 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:16:27.928731 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:16:27.928734 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:16:27.928993 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:16:27.929139 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0620 20:16:27.929821 (+ 682us) server_negotiation.cc:299] Negotiation successful
0620 20:16:27.930053 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":364,"thread_start_us":283,"threads_started":1}
I20260620 20:16:27.931368 23078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:27.911906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:44239 (local address 127.16.217.1:55091)
0620 20:16:27.912631 (+ 725us) negotiation.cc:107] Waiting for socket to connect
0620 20:16:27.912674 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 20:16:27.914464 (+ 1790us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:16:27.915181 (+ 717us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:16:27.915189 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:16:27.915679 (+ 490us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:16:27.916234 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:16:27.916249 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:16:27.917250 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:16:27.917261 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:16:27.918653 (+ 1392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:16:27.918664 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:16:27.919572 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:16:27.919582 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:16:27.919936 (+ 354us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:27.920812 (+ 876us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:16:27.920842 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:16:27.923806 (+ 2964us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:16:27.927614 (+ 3808us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:16:27.927623 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:16:27.927640 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:16:27.928055 (+ 415us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:16:27.928409 (+ 354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:16:27.928415 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:16:27.928419 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:16:27.928596 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:16:27.929113 (+ 517us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:16:27.929121 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:16:27.929454 (+ 333us) client_negotiation.cc:769] Sending connection context
0620 20:16:27.929709 (+ 255us) client_negotiation.cc:240] Negotiation successful
0620 20:16:27.930256 (+ 547us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":507,"thread_start_us":298,"threads_started":1}
I20260620 20:16:27.932986 23075 heartbeater.cc:347] Connected to a master server at 127.16.217.62:44239
I20260620 20:16:27.933295 23075 heartbeater.cc:464] Registering TS with master...
I20260620 20:16:27.934055 23075 heartbeater.cc:511] Master 127.16.217.62:44239 requested a full tablet report, sending...
I20260620 20:16:27.936074 22886 ts_manager.cc:194] Registered new tserver with Master: 4fd4d1a935514a81bb53d328f066b287 (127.16.217.1:35279)
I20260620 20:16:27.937462 22886 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.217.1@KRBTEST.COM'} at 127.16.217.1:55091
WARNING: no policy specified for kudu/127.16.217.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.217.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.217.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.2.keytab.
Entry for principal kudu/127.16.217.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.2.keytab.
I20260620 20:16:27.966315 17252 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
/tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986536687152-17252-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986536687152-17252-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986536687152-17252-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986536687152-17252-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.217.2:0
--local_ip_for_outbound_sockets=127.16.217.2
--webserver_interface=127.16.217.2
--webserver_port=0
--tserver_master_addrs=127.16.217.62:44239
--keytab_file=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.2.keytab
--principal=kudu/127.16.217.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.217.20:42447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986536687152-17252-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986536687152-17252-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986536687152-17252-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:16:28.284384 23081 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:16:28.284696 23081 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:16:28.284870 23081 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:16:28.292001 23081 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:16:28.292157 23081 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:16:28.292340 23081 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.217.2
I20260620 20:16:28.302173 23081 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.217.20:42447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986536687152-17252-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986536687152-17252-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.2.keytab
--principal=kudu/127.16.217.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.217.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986536687152-17252-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.217.2
--webserver_port=0
--tserver_master_addrs=127.16.217.62:44239
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.217.2
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986536687152-17252-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:16:28.303560 23081 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:16:28.304867 23081 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:16:28.317963 23086 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:16:28.317907 23087 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:16:28.317940 23089 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:16:28.319065 23081 server_base.cc:1034] running on GCE node
I20260620 20:16:28.319674 23081 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:16:28.320667 23081 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:16:28.321942 23081 hybrid_clock.cc:648] HybridClock initialized: now 1781986588321884 us; error 47 us; skew 500 ppm
I20260620 20:16:28.322263 23081 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:16:28 dist-test-slave-2hpg krb5kdc[22841](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986588, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.217.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:16:28.328563 23081 init.cc:377] Logged in from keytab as kudu/127.16.217.2@KRBTEST.COM (short username kudu)
I20260620 20:16:28.330138 23081 webserver.cc:460] Webserver started at http://127.16.217.2:36303/ using document root <none> and password file <none>
I20260620 20:16:28.330824 23081 fs_manager.cc:359] Metadata directory not provided
I20260620 20:16:28.330964 23081 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:16:28.331267 23081 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:16:28.335053 23081 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986536687152-17252-0/minicluster-data/ts-1/data/instance:
uuid: "5fdc1c9dc94c48c891da2a5ff05b057b"
format_stamp: "Formatted at 2026-06-20 20:16:28 on dist-test-slave-2hpg"
server_key: "4eec689c792fe82b6f97df3b816d442f"
server_key_iv: "e1e2241f747ef7ad0d55b2e2b4aad163"
server_key_version: "encryptionkey@0"
I20260620 20:16:28.336010 23081 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986536687152-17252-0/minicluster-data/ts-1/wal/instance:
uuid: "5fdc1c9dc94c48c891da2a5ff05b057b"
format_stamp: "Formatted at 2026-06-20 20:16:28 on dist-test-slave-2hpg"
server_key: "4eec689c792fe82b6f97df3b816d442f"
server_key_iv: "e1e2241f747ef7ad0d55b2e2b4aad163"
server_key_version: "encryptionkey@0"
I20260620 20:16:28.342288 23081 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.005s
I20260620 20:16:28.346428 23096 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:16:28.348068 23081 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 20:16:28.348229 23081 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986536687152-17252-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986536687152-17252-0/minicluster-data/ts-1/wal
uuid: "5fdc1c9dc94c48c891da2a5ff05b057b"
format_stamp: "Formatted at 2026-06-20 20:16:28 on dist-test-slave-2hpg"
server_key: "4eec689c792fe82b6f97df3b816d442f"
server_key_iv: "e1e2241f747ef7ad0d55b2e2b4aad163"
server_key_version: "encryptionkey@0"
I20260620 20:16:28.348459 23081 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986536687152-17252-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986536687152-17252-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986536687152-17252-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:16:28.383586 23081 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:16:28.388120 23081 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:16:28.388404 23081 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:16:28.389365 23081 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:16:28.390841 23081 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:16:28.390921 23081 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:16:28.391027 23081 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:16:28.391077 23081 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:16:28.427027 23081 rpc_server.cc:307] RPC server started. Bound to: 127.16.217.2:35579
I20260620 20:16:28.427064 23208 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.217.2:35579 every 8 connection(s)
I20260620 20:16:28.428422 23081 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986536687152-17252-0/minicluster-data/ts-1/data/info.pb
I20260620 20:16:28.435613 17252 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu as pid 23081
I20260620 20:16:28.435833 17252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986536687152-17252-0/minicluster-data/ts-1/wal/instance
I20260620 20:16:28.436257 17252 external_mini_cluster.cc:1428] Setting key 64c642b65305c20145bdf511ab476e05
Jun 20 20:16:28 dist-test-slave-2hpg krb5kdc[22841](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986588, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.217.2@KRBTEST.COM for kudu/127.16.217.62@KRBTEST.COM
I20260620 20:16:28.450908 23211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:28.431075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.217.2:48993 (local address 127.16.217.62:44239)
0620 20:16:28.431503 (+ 428us) server_negotiation.cc:206] Beginning negotiation
0620 20:16:28.431507 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:16:28.432456 (+ 949us) server_negotiation.cc:407] Connection header received
0620 20:16:28.434315 (+ 1859us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:16:28.434320 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:16:28.434416 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:16:28.434533 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:16:28.436067 (+ 1534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:28.436934 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:28.438914 (+ 1980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:28.439334 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:28.439469 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:28.443408 (+ 3939us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:16:28.443412 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:16:28.443416 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:16:28.443445 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:16:28.446886 (+ 3441us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:16:28.448014 (+ 1128us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:16:28.448022 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:16:28.448029 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:16:28.448200 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:16:28.448687 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:16:28.448695 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:16:28.448701 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:16:28.449082 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:16:28.449241 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0620 20:16:28.450133 (+ 892us) server_negotiation.cc:299] Negotiation successful
0620 20:16:28.450394 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":318,"thread_start_us":229,"threads_started":1}
I20260620 20:16:28.451566 23212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:28.431520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:44239 (local address 127.16.217.2:48993)
0620 20:16:28.432297 (+ 777us) negotiation.cc:107] Waiting for socket to connect
0620 20:16:28.432340 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 20:16:28.434054 (+ 1714us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:16:28.434771 (+ 717us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:16:28.434781 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:16:28.435266 (+ 485us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:16:28.435878 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:16:28.435896 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:16:28.437084 (+ 1188us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:16:28.437089 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:16:28.438778 (+ 1689us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:16:28.438789 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:16:28.439554 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:16:28.439561 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:16:28.439800 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:28.440482 (+ 682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:16:28.440508 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:16:28.443189 (+ 2681us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:16:28.447081 (+ 3892us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:16:28.447092 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:16:28.447111 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:16:28.447868 (+ 757us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:16:28.448323 (+ 455us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:16:28.448326 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:16:28.448329 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:16:28.448559 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:16:28.449212 (+ 653us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:16:28.449219 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:16:28.449667 (+ 448us) client_negotiation.cc:769] Sending connection context
0620 20:16:28.450004 (+ 337us) client_negotiation.cc:240] Negotiation successful
0620 20:16:28.450413 (+ 409us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":543,"thread_start_us":342,"threads_started":1}
I20260620 20:16:28.453152 23209 heartbeater.cc:347] Connected to a master server at 127.16.217.62:44239
I20260620 20:16:28.453485 23209 heartbeater.cc:464] Registering TS with master...
I20260620 20:16:28.454401 23209 heartbeater.cc:511] Master 127.16.217.62:44239 requested a full tablet report, sending...
I20260620 20:16:28.455912 22886 ts_manager.cc:194] Registered new tserver with Master: 5fdc1c9dc94c48c891da2a5ff05b057b (127.16.217.2:35579)
I20260620 20:16:28.456737 22886 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.217.2@KRBTEST.COM'} at 127.16.217.2:48993
WARNING: no policy specified for kudu/127.16.217.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.217.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.217.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.3.keytab.
Entry for principal kudu/127.16.217.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.3.keytab.
I20260620 20:16:28.484647 17252 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
/tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986536687152-17252-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986536687152-17252-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986536687152-17252-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986536687152-17252-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.217.3:0
--local_ip_for_outbound_sockets=127.16.217.3
--webserver_interface=127.16.217.3
--webserver_port=0
--tserver_master_addrs=127.16.217.62:44239
--keytab_file=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.3.keytab
--principal=kudu/127.16.217.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.217.20:42447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986536687152-17252-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986536687152-17252-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986536687152-17252-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:16:28.807292 23215 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:16:28.807636 23215 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:16:28.807806 23215 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:16:28.815315 23215 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:16:28.815478 23215 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:16:28.815671 23215 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.217.3
I20260620 20:16:28.825711 23215 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.217.20:42447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986536687152-17252-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986536687152-17252-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.3.keytab
--principal=kudu/127.16.217.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.217.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986536687152-17252-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.217.3
--webserver_port=0
--tserver_master_addrs=127.16.217.62:44239
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.217.3
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986536687152-17252-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:16:28.827075 23215 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:16:28.828363 23215 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:16:28.841436 23220 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:16:28.841499 23223 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:16:28.841656 23221 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:16:28.842613 23215 server_base.cc:1034] running on GCE node
I20260620 20:16:28.843214 23215 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:16:28.844142 23215 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:16:28.845369 23215 hybrid_clock.cc:648] HybridClock initialized: now 1781986588845327 us; error 37 us; skew 500 ppm
I20260620 20:16:28.845688 23215 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:16:28 dist-test-slave-2hpg krb5kdc[22841](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986588, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.217.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:16:28.848846 23215 init.cc:377] Logged in from keytab as kudu/127.16.217.3@KRBTEST.COM (short username kudu)
I20260620 20:16:28.850306 23215 webserver.cc:460] Webserver started at http://127.16.217.3:36627/ using document root <none> and password file <none>
I20260620 20:16:28.850939 23215 fs_manager.cc:359] Metadata directory not provided
I20260620 20:16:28.851037 23215 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:16:28.851289 23215 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:16:28.854444 23215 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986536687152-17252-0/minicluster-data/ts-2/data/instance:
uuid: "f196ee29cf6f48e7b394619766587d7d"
format_stamp: "Formatted at 2026-06-20 20:16:28 on dist-test-slave-2hpg"
server_key: "7cd67ad828690b956d1da137a1276354"
server_key_iv: "dde5c5a9b88b15d985c21df6c0e388b9"
server_key_version: "encryptionkey@0"
I20260620 20:16:28.855201 23215 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986536687152-17252-0/minicluster-data/ts-2/wal/instance:
uuid: "f196ee29cf6f48e7b394619766587d7d"
format_stamp: "Formatted at 2026-06-20 20:16:28 on dist-test-slave-2hpg"
server_key: "7cd67ad828690b956d1da137a1276354"
server_key_iv: "dde5c5a9b88b15d985c21df6c0e388b9"
server_key_version: "encryptionkey@0"
I20260620 20:16:28.860245 23215 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260620 20:16:28.863973 23230 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:16:28.865372 23215 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 20:16:28.865510 23215 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986536687152-17252-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986536687152-17252-0/minicluster-data/ts-2/wal
uuid: "f196ee29cf6f48e7b394619766587d7d"
format_stamp: "Formatted at 2026-06-20 20:16:28 on dist-test-slave-2hpg"
server_key: "7cd67ad828690b956d1da137a1276354"
server_key_iv: "dde5c5a9b88b15d985c21df6c0e388b9"
server_key_version: "encryptionkey@0"
I20260620 20:16:28.865684 23215 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986536687152-17252-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986536687152-17252-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986536687152-17252-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:16:28.879223 23215 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:16:28.883821 23215 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:16:28.884061 23215 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:16:28.884982 23215 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:16:28.886364 23215 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:16:28.886436 23215 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:16:28.886519 23215 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:16:28.886566 23215 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:16:28.918365 23215 rpc_server.cc:307] RPC server started. Bound to: 127.16.217.3:45309
I20260620 20:16:28.918401 23342 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.217.3:45309 every 8 connection(s)
I20260620 20:16:28.919687 23215 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986536687152-17252-0/minicluster-data/ts-2/data/info.pb
I20260620 20:16:28.924405 17252 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu as pid 23215
I20260620 20:16:28.924589 17252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986536687152-17252-0/minicluster-data/ts-2/wal/instance
I20260620 20:16:28.925020 17252 external_mini_cluster.cc:1428] Setting key 56fc50f2024321bf47378b1d8b0d497e
Jun 20 20:16:28 dist-test-slave-2hpg krb5kdc[22841](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986588, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.217.3@KRBTEST.COM for kudu/127.16.217.62@KRBTEST.COM
I20260620 20:16:28.940380 23075 heartbeater.cc:503] Master 127.16.217.62:44239 was elected leader, sending a full tablet report...
I20260620 20:16:28.940487 23211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:28.922327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.217.3:57511 (local address 127.16.217.62:44239)
0620 20:16:28.922533 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0620 20:16:28.922539 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:16:28.923704 (+ 1165us) server_negotiation.cc:407] Connection header received
0620 20:16:28.925649 (+ 1945us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:16:28.925654 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:16:28.925768 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:16:28.925909 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:16:28.928183 (+ 2274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:28.928971 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:28.930657 (+ 1686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:28.931089 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:28.931186 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:28.934674 (+ 3488us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:16:28.934679 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:16:28.934682 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:16:28.934711 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:16:28.937138 (+ 2427us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:16:28.937966 (+ 828us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:16:28.937971 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:16:28.937973 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:16:28.938084 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:16:28.938520 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:16:28.938525 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:16:28.938529 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:16:28.938869 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:16:28.939033 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0620 20:16:28.939900 (+ 867us) server_negotiation.cc:299] Negotiation successful
0620 20:16:28.940164 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 20:16:28.941752 23345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:28.922775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:44239 (local address 127.16.217.3:57511)
0620 20:16:28.923524 (+ 749us) negotiation.cc:107] Waiting for socket to connect
0620 20:16:28.923560 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 20:16:28.925337 (+ 1777us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:16:28.926267 (+ 930us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:16:28.926279 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:16:28.927045 (+ 766us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:16:28.927942 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:16:28.927970 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:16:28.929112 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:16:28.929116 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:16:28.930512 (+ 1396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:16:28.930525 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:16:28.931284 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:16:28.931290 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:16:28.931493 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:28.932251 (+ 758us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:16:28.932275 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:16:28.934476 (+ 2201us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:16:28.937292 (+ 2816us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:16:28.937299 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:16:28.937312 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:16:28.937790 (+ 478us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:16:28.938177 (+ 387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:16:28.938180 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:16:28.938183 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:16:28.938392 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:16:28.938993 (+ 601us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:16:28.938999 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:16:28.939466 (+ 467us) client_negotiation.cc:769] Sending connection context
0620 20:16:28.939766 (+ 300us) client_negotiation.cc:240] Negotiation successful
0620 20:16:28.940314 (+ 548us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":526,"thread_start_us":322,"threads_started":1}
I20260620 20:16:28.943148 23343 heartbeater.cc:347] Connected to a master server at 127.16.217.62:44239
I20260620 20:16:28.943454 23343 heartbeater.cc:464] Registering TS with master...
I20260620 20:16:28.944147 23343 heartbeater.cc:511] Master 127.16.217.62:44239 requested a full tablet report, sending...
I20260620 20:16:28.945712 22886 ts_manager.cc:194] Registered new tserver with Master: f196ee29cf6f48e7b394619766587d7d (127.16.217.3:45309)
I20260620 20:16:28.946841 22886 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.217.3@KRBTEST.COM'} at 127.16.217.3:57511
I20260620 20:16:28.953150 17252 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986536687152-17252-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986536687152-17252-0/krb5kdc/test-admin.keytab.
Jun 20 20:16:28 dist-test-slave-2hpg krb5kdc[22841](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986588, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:16:28.980041 17252 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260620 20:16:28.985347 23350 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:16:28.982844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39210 (local address 127.16.217.1:35279)
0620 20:16:28.983445 (+ 601us) server_negotiation.cc:206] Beginning negotiation
0620 20:16:28.983456 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 20:16:28.983491 (+ 35us) server_negotiation.cc:407] Connection header received
0620 20:16:28.983568 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:16:28.983576 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:16:28.983796 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:16:28.983963 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:16:28.984890 (+ 927us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39210: BlockingRecv error: recv got EOF from 127.0.0.1:39210 (error 108)
Metrics: {"server-negotiator.queue_time_us":430,"thread_start_us":259,"threads_started":1}
W20260620 20:16:28.985446 17252 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986536687152-17252-0/krb5kdc/krb5cc' of type FILE
Jun 20 20:16:28 dist-test-slave-2hpg krb5kdc[22841](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986588, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:16:28.987053 17252 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 20 20:16:28 dist-test-slave-2hpg krb5kdc[22841](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986588, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:16:28.988468 17252 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 20 20:16:28 dist-test-slave-2hpg krb5kdc[22841](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986588, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.217.1@KRBTEST.COM
I20260620 20:16:29.004412 23350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:28.990245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39214 (local address 127.16.217.1:35279)
0620 20:16:28.990468 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0620 20:16:28.990474 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:16:28.990727 (+ 253us) server_negotiation.cc:407] Connection header received
0620 20:16:28.990852 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:16:28.990856 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:16:28.990917 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:16:28.991006 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:16:28.991969 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:28.993012 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:28.994608 (+ 1596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:28.995210 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:28.995353 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:28.999050 (+ 3697us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:16:28.999062 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:16:28.999077 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:16:28.999110 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:16:29.001667 (+ 2557us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:16:29.002411 (+ 744us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:16:29.002419 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:16:29.002421 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:16:29.002534 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:16:29.002902 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:16:29.002906 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:16:29.002908 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:16:29.003242 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:16:29.003540 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0620 20:16:29.003878 (+ 338us) server_negotiation.cc:299] Negotiation successful
0620 20:16:29.004118 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
W20260620 20:16:29.010710 23350 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:16:29.008759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39222 (local address 127.16.217.1:35279)
0620 20:16:29.008979 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0620 20:16:29.008986 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:16:29.009168 (+ 182us) server_negotiation.cc:407] Connection header received
0620 20:16:29.009320 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:16:29.009323 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:16:29.009387 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:16:29.009469 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:16:29.010461 (+ 992us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39222: BlockingRecv error: recv got EOF from 127.0.0.1:39222 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
W20260620 20:16:29.010916 17252 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986536687152-17252-0/krb5kdc/krb5cc' of type FILE
Jun 20 20:16:29 dist-test-slave-2hpg krb5kdc[22841](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986589, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:16:29.012455 17252 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 20 20:16:29 dist-test-slave-2hpg krb5kdc[22841](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986589, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:16:29.013743 17252 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 20 20:16:29 dist-test-slave-2hpg krb5kdc[22841](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986589, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.217.1@KRBTEST.COM
I20260620 20:16:29.027873 23350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:29.015155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39230 (local address 127.16.217.1:35279)
0620 20:16:29.015358 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0620 20:16:29.015364 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:16:29.015601 (+ 237us) server_negotiation.cc:407] Connection header received
0620 20:16:29.015782 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:16:29.015788 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:16:29.015862 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:16:29.015965 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:16:29.016894 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:29.017621 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:29.018893 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:29.019354 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:29.019492 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:29.022949 (+ 3457us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:16:29.022954 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:16:29.022958 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:16:29.023002 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:16:29.025393 (+ 2391us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:16:29.026064 (+ 671us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:16:29.026076 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:16:29.026079 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:16:29.026193 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:16:29.026579 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:16:29.026584 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:16:29.026587 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:16:29.026856 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:16:29.027111 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0620 20:16:29.027386 (+ 275us) server_negotiation.cc:299] Negotiation successful
0620 20:16:29.027588 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
W20260620 20:16:29.032232 23350 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:16:29.030630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39244 (local address 127.16.217.1:35279)
0620 20:16:29.030818 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0620 20:16:29.030824 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:16:29.030943 (+ 119us) server_negotiation.cc:407] Connection header received
0620 20:16:29.031109 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:16:29.031115 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:16:29.031192 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:16:29.031288 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:16:29.032002 (+ 714us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39244: BlockingRecv error: recv got EOF from 127.0.0.1:39244 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
W20260620 20:16:29.032375 17252 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986536687152-17252-0/krb5kdc/krb5cc' of type FILE
Jun 20 20:16:29 dist-test-slave-2hpg krb5kdc[22841](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986589, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:16:29.033852 17252 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 20 20:16:29 dist-test-slave-2hpg krb5kdc[22841](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986589, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:16:29.035192 17252 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 20 20:16:29 dist-test-slave-2hpg krb5kdc[22841](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986589, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.217.1@KRBTEST.COM
I20260620 20:16:29.048804 23350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:29.036391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39250 (local address 127.16.217.1:35279)
0620 20:16:29.036588 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0620 20:16:29.036594 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:16:29.036775 (+ 181us) server_negotiation.cc:407] Connection header received
0620 20:16:29.036863 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:16:29.036867 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:16:29.036931 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:16:29.037022 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:16:29.037953 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:29.038696 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:29.039833 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:29.040258 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:29.040363 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:29.043856 (+ 3493us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:16:29.043861 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:16:29.043864 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:16:29.043894 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:16:29.046402 (+ 2508us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:16:29.046985 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:16:29.046989 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:16:29.046991 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:16:29.047131 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:16:29.047469 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:16:29.047473 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:16:29.047475 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:16:29.047798 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:16:29.048024 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0620 20:16:29.048247 (+ 223us) server_negotiation.cc:299] Negotiation successful
0620 20:16:29.048407 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260620 20:16:29.050375 17252 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu pid 22947
I20260620 20:16:29.061902 23350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:29.051021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39258 (local address 127.16.217.1:35279)
0620 20:16:29.051228 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0620 20:16:29.051233 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:16:29.051431 (+ 198us) server_negotiation.cc:407] Connection header received
0620 20:16:29.051552 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:16:29.051556 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:16:29.051611 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:16:29.051691 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:16:29.052639 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:29.053373 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:29.054697 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:29.055148 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:29.055247 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:29.057171 (+ 1924us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:16:29.057176 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:16:29.057179 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:16:29.057207 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:16:29.059513 (+ 2306us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:16:29.060054 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:16:29.060058 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:16:29.060060 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:16:29.060192 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:16:29.060533 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:16:29.060537 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:16:29.060539 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:16:29.060816 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:16:29.061019 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0620 20:16:29.061291 (+ 272us) server_negotiation.cc:299] Negotiation successful
0620 20:16:29.061523 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 20:16:29.062556 23070 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:16:29.144985 17252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu with pid 22947
I20260620 20:16:29.161026 17252 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu pid 23081
Jun 20 20:16:29 dist-test-slave-2hpg krb5kdc[22841](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986589, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.217.2@KRBTEST.COM
I20260620 20:16:29.176904 23363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:29.162150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38490 (local address 127.16.217.2:35579)
0620 20:16:29.162680 (+ 530us) server_negotiation.cc:206] Beginning negotiation
0620 20:16:29.162688 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 20:16:29.162723 (+ 35us) server_negotiation.cc:407] Connection header received
0620 20:16:29.162787 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:16:29.162793 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:16:29.163003 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:16:29.163145 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:16:29.164067 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:29.165200 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:29.166637 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:29.167236 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:29.167349 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:29.171263 (+ 3914us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:16:29.171274 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:16:29.171288 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:16:29.171325 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:16:29.174213 (+ 2888us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:16:29.174896 (+ 683us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:16:29.174903 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:16:29.174905 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:16:29.175021 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:16:29.175424 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:16:29.175428 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:16:29.175430 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:16:29.175756 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:16:29.176037 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0620 20:16:29.176331 (+ 294us) server_negotiation.cc:299] Negotiation successful
0620 20:16:29.176589 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":395,"thread_start_us":257,"threads_started":1}
I20260620 20:16:29.178228 23204 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:16:29.261752 17252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu with pid 23081
I20260620 20:16:29.277570 17252 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu pid 23215
Jun 20 20:16:29 dist-test-slave-2hpg krb5kdc[22841](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986589, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.217.3@KRBTEST.COM
I20260620 20:16:29.293282 23365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:29.278696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48864 (local address 127.16.217.3:45309)
0620 20:16:29.279336 (+ 640us) server_negotiation.cc:206] Beginning negotiation
0620 20:16:29.279343 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:16:29.279373 (+ 30us) server_negotiation.cc:407] Connection header received
0620 20:16:29.279432 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:16:29.279438 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:16:29.279713 (+ 275us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:16:29.279854 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:16:29.280921 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:29.282061 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:29.283336 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:29.283876 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:29.283972 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:29.287910 (+ 3938us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:16:29.287921 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:16:29.287936 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:16:29.287969 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:16:29.290783 (+ 2814us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:16:29.291440 (+ 657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:16:29.291447 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:16:29.291450 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:16:29.291556 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:16:29.291853 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:16:29.291856 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:16:29.291858 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:16:29.292202 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:16:29.292419 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0620 20:16:29.292664 (+ 245us) server_negotiation.cc:299] Negotiation successful
0620 20:16:29.292969 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":484,"thread_start_us":318,"threads_started":1}
I20260620 20:16:29.294701 23338 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:16:29.378127 17252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu with pid 23215
I20260620 20:16:29.393616 17252 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu pid 22854
I20260620 20:16:29.394924 22916 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:16:29.495645 17252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu with pid 22854
2026-06-20T20:16:29Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (3867 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986536687152-17252-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 20:16:29 dist-test-slave-2hpg krb5kdc[23370](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 20:16:29 dist-test-slave-2hpg krb5kdc[23370](info): set up 2 sockets
Jun 20 20:16:29 dist-test-slave-2hpg krb5kdc[23370](info): commencing operation
krb5kdc: starting...
W20260620 20:16:30.590834 17252 mini_kdc.cc:121] Time spent starting KDC: real 1.053s user 0.005s sys 0.013s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986536687152-17252-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986536687152-17252-0/krb5kdc/test-user.keytab.
Jun 20 20:16:30 dist-test-slave-2hpg krb5kdc[23370](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986590, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T20:16:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T20:16:30Z Disabled control of system clock
WARNING: no policy specified for oryx/127.16.217.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.16.217.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.16.217.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986536687152-17252-0/krb5kdc/oryx_127.16.217.62.keytab.
Entry for principal oryx/127.16.217.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986536687152-17252-0/krb5kdc/oryx_127.16.217.62.keytab.
I20260620 20:16:30.795236 17252 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
/tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986536687152-17252-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986536687152-17252-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986536687152-17252-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986536687152-17252-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.217.62:36319
--webserver_interface=127.16.217.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986536687152-17252-0/krb5kdc/oryx_127.16.217.62.keytab
--principal=oryx/127.16.217.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.217.20:38255
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.217.62:36319
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986536687152-17252-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986536687152-17252-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986536687152-17252-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:16:31.114722 23383 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:16:31.115123 23383 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:16:31.115284 23383 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:16:31.122495 23383 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:16:31.122598 23383 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:16:31.122644 23383 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:16:31.122757 23383 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:16:31.122820 23383 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:16:31.132464 23383 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.217.20:38255
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986536687152-17252-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986536687152-17252-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.217.62:36319
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986536687152-17252-0/krb5kdc/oryx_127.16.217.62.keytab
--principal=oryx/127.16.217.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.217.62:36319
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986536687152-17252-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.217.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986536687152-17252-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:16:31.133813 23383 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:16:31.135149 23383 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:16:31.146989 23388 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:16:31.146921 23391 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:16:31.146967 23389 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:16:31.148020 23383 server_base.cc:1034] running on GCE node
I20260620 20:16:31.148806 23383 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:16:31.150286 23383 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:16:31.151532 23383 hybrid_clock.cc:648] HybridClock initialized: now 1781986591151483 us; error 60 us; skew 500 ppm
I20260620 20:16:31.151885 23383 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:16:31 dist-test-slave-2hpg krb5kdc[23370](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986591, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.217.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:16:31.155140 23383 init.cc:377] Logged in from keytab as oryx/127.16.217.62@KRBTEST.COM (short username oryx)
I20260620 20:16:31.156610 23383 webserver.cc:460] Webserver started at http://127.16.217.62:39031/ using document root <none> and password file <none>
I20260620 20:16:31.157303 23383 fs_manager.cc:359] Metadata directory not provided
I20260620 20:16:31.157449 23383 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:16:31.157743 23383 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:16:31.160710 23383 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986536687152-17252-0/minicluster-data/master-0/data/instance:
uuid: "358753b177ca4441ba3b33b1e5fd7d66"
format_stamp: "Formatted at 2026-06-20 20:16:31 on dist-test-slave-2hpg"
server_key: "e632137743b4623d5a4ce28e87edc8fd"
server_key_iv: "9c8c4e412e0749f968ad62f7565ecf29"
server_key_version: "encryptionkey@0"
I20260620 20:16:31.161497 23383 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986536687152-17252-0/minicluster-data/master-0/wal/instance:
uuid: "358753b177ca4441ba3b33b1e5fd7d66"
format_stamp: "Formatted at 2026-06-20 20:16:31 on dist-test-slave-2hpg"
server_key: "e632137743b4623d5a4ce28e87edc8fd"
server_key_iv: "9c8c4e412e0749f968ad62f7565ecf29"
server_key_version: "encryptionkey@0"
I20260620 20:16:31.166673 23383 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260620 20:16:31.170096 23398 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:16:31.171469 23383 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 20:16:31.171648 23383 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986536687152-17252-0/minicluster-data/master-0/data,/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986536687152-17252-0/minicluster-data/master-0/wal
uuid: "358753b177ca4441ba3b33b1e5fd7d66"
format_stamp: "Formatted at 2026-06-20 20:16:31 on dist-test-slave-2hpg"
server_key: "e632137743b4623d5a4ce28e87edc8fd"
server_key_iv: "9c8c4e412e0749f968ad62f7565ecf29"
server_key_version: "encryptionkey@0"
I20260620 20:16:31.171849 23383 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986536687152-17252-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986536687152-17252-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986536687152-17252-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:16:31.199299 23383 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:16:31.203876 23383 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:16:31.204170 23383 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:16:31.222415 23383 rpc_server.cc:307] RPC server started. Bound to: 127.16.217.62:36319
I20260620 20:16:31.222445 23449 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.217.62:36319 every 8 connection(s)
I20260620 20:16:31.223804 23383 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986536687152-17252-0/minicluster-data/master-0/data/info.pb
I20260620 20:16:31.224922 17252 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu as pid 23383
I20260620 20:16:31.225136 17252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986536687152-17252-0/minicluster-data/master-0/wal/instance
I20260620 20:16:31.225793 17252 external_mini_cluster.cc:1428] Setting key cc18395d699e48177066c8a4adc7e2d7
I20260620 20:16:31.228207 23450 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 20 20:16:31 dist-test-slave-2hpg krb5kdc[23370](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986590, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.16.217.62@KRBTEST.COM
I20260620 20:16:31.241012 23450 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 358753b177ca4441ba3b33b1e5fd7d66: Bootstrap starting.
I20260620 20:16:31.245193 23450 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 358753b177ca4441ba3b33b1e5fd7d66: Neither blocks nor log segments found. Creating new log.
I20260620 20:16:31.246449 23450 log.cc:824] T 00000000000000000000000000000000 P 358753b177ca4441ba3b33b1e5fd7d66: Log is configured to *not* fsync() on all Append() calls
I20260620 20:16:31.249997 23450 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 358753b177ca4441ba3b33b1e5fd7d66: No bootstrap required, opened a new log
I20260620 20:16:31.250784 23453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:31.227843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50246 (local address 127.16.217.62:36319)
0620 20:16:31.228695 (+ 852us) server_negotiation.cc:206] Beginning negotiation
0620 20:16:31.228711 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 20:16:31.228771 (+ 60us) server_negotiation.cc:407] Connection header received
0620 20:16:31.231009 (+ 2238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:16:31.231036 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:16:31.231552 (+ 516us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:16:31.232025 (+ 473us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:16:31.233338 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:31.235411 (+ 2073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:31.236610 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:31.237653 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:31.237894 (+ 241us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:31.241806 (+ 3912us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:16:31.241817 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:16:31.241836 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:16:31.241911 (+ 75us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:16:31.245747 (+ 3836us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:16:31.246786 (+ 1039us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:16:31.246795 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:16:31.246803 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:16:31.247028 (+ 225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:16:31.247535 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:16:31.247538 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:16:31.247540 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:16:31.248290 (+ 750us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:16:31.248571 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0620 20:16:31.248951 (+ 380us) server_negotiation.cc:299] Negotiation successful
0620 20:16:31.249397 (+ 446us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":610,"thread_start_us":386,"threads_started":1}
I20260620 20:16:31.257710 23450 raft_consensus.cc:348] T 00000000000000000000000000000000 P 358753b177ca4441ba3b33b1e5fd7d66 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "358753b177ca4441ba3b33b1e5fd7d66" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 36319 } }
I20260620 20:16:31.258144 23450 raft_consensus.cc:374] T 00000000000000000000000000000000 P 358753b177ca4441ba3b33b1e5fd7d66 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:16:31.258224 23450 raft_consensus.cc:729] T 00000000000000000000000000000000 P 358753b177ca4441ba3b33b1e5fd7d66 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 358753b177ca4441ba3b33b1e5fd7d66, State: Initialized, Role: FOLLOWER
I20260620 20:16:31.258718 23450 consensus_queue.cc:260] T 00000000000000000000000000000000 P 358753b177ca4441ba3b33b1e5fd7d66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "358753b177ca4441ba3b33b1e5fd7d66" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 36319 } }
I20260620 20:16:31.258903 23450 raft_consensus.cc:388] T 00000000000000000000000000000000 P 358753b177ca4441ba3b33b1e5fd7d66 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 20:16:31.259019 23450 raft_consensus.cc:482] T 00000000000000000000000000000000 P 358753b177ca4441ba3b33b1e5fd7d66 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 20:16:31.259191 23450 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 358753b177ca4441ba3b33b1e5fd7d66 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:16:31.261598 23450 raft_consensus.cc:504] T 00000000000000000000000000000000 P 358753b177ca4441ba3b33b1e5fd7d66 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "358753b177ca4441ba3b33b1e5fd7d66" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 36319 } }
I20260620 20:16:31.262111 23450 leader_election.cc:302] T 00000000000000000000000000000000 P 358753b177ca4441ba3b33b1e5fd7d66 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 358753b177ca4441ba3b33b1e5fd7d66; no voters:
I20260620 20:16:31.262652 23450 leader_election.cc:288] T 00000000000000000000000000000000 P 358753b177ca4441ba3b33b1e5fd7d66 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 20:16:31.262835 23455 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 358753b177ca4441ba3b33b1e5fd7d66 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:16:31.263499 23455 raft_consensus.cc:686] T 00000000000000000000000000000000 P 358753b177ca4441ba3b33b1e5fd7d66 [term 1 LEADER]: Becoming Leader. State: Replica: 358753b177ca4441ba3b33b1e5fd7d66, State: Running, Role: LEADER
I20260620 20:16:31.264025 23455 consensus_queue.cc:237] T 00000000000000000000000000000000 P 358753b177ca4441ba3b33b1e5fd7d66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "358753b177ca4441ba3b33b1e5fd7d66" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 36319 } }
I20260620 20:16:31.264818 23450 sys_catalog.cc:564] T 00000000000000000000000000000000 P 358753b177ca4441ba3b33b1e5fd7d66 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:16:31.267201 23457 sys_catalog.cc:455] T 00000000000000000000000000000000 P 358753b177ca4441ba3b33b1e5fd7d66 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 358753b177ca4441ba3b33b1e5fd7d66. Latest consensus state: current_term: 1 leader_uuid: "358753b177ca4441ba3b33b1e5fd7d66" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "358753b177ca4441ba3b33b1e5fd7d66" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 36319 } } }
I20260620 20:16:31.267590 23457 sys_catalog.cc:458] T 00000000000000000000000000000000 P 358753b177ca4441ba3b33b1e5fd7d66 [sys.catalog]: This master's current role is: LEADER
I20260620 20:16:31.267189 23456 sys_catalog.cc:455] T 00000000000000000000000000000000 P 358753b177ca4441ba3b33b1e5fd7d66 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "358753b177ca4441ba3b33b1e5fd7d66" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "358753b177ca4441ba3b33b1e5fd7d66" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 36319 } } }
I20260620 20:16:31.267843 23456 sys_catalog.cc:458] T 00000000000000000000000000000000 P 358753b177ca4441ba3b33b1e5fd7d66 [sys.catalog]: This master's current role is: LEADER
I20260620 20:16:31.269111 23463 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 20:16:31.274549 23463 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 20:16:31.283748 23463 catalog_manager.cc:1347] Generated new cluster ID: a002c6e7a28743d2b82d56e2d2592ade
I20260620 20:16:31.283891 23463 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 20:16:31.299185 23463 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 20:16:31.300513 23463 catalog_manager.cc:1504] Loading token signing keys...
I20260620 20:16:31.308513 23463 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 358753b177ca4441ba3b33b1e5fd7d66: Generated new TSK 0
I20260620 20:16:31.309590 23463 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.16.217.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.16.217.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.16.217.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986536687152-17252-0/krb5kdc/oryx_127.16.217.1.keytab.
Entry for principal oryx/127.16.217.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986536687152-17252-0/krb5kdc/oryx_127.16.217.1.keytab.
I20260620 20:16:31.364995 17252 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
/tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986536687152-17252-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986536687152-17252-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986536687152-17252-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986536687152-17252-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.217.1:0
--local_ip_for_outbound_sockets=127.16.217.1
--webserver_interface=127.16.217.1
--webserver_port=0
--tserver_master_addrs=127.16.217.62:36319
--keytab_file=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986536687152-17252-0/krb5kdc/oryx_127.16.217.1.keytab
--principal=oryx/127.16.217.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.217.20:38255
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986536687152-17252-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986536687152-17252-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986536687152-17252-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:16:31.736613 23476 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:16:31.736943 23476 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:16:31.737119 23476 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:16:31.744235 23476 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:16:31.744388 23476 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:16:31.744577 23476 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.217.1
I20260620 20:16:31.754259 23476 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.217.20:38255
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986536687152-17252-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986536687152-17252-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986536687152-17252-0/krb5kdc/oryx_127.16.217.1.keytab
--principal=oryx/127.16.217.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.217.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986536687152-17252-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.217.1
--webserver_port=0
--tserver_master_addrs=127.16.217.62:36319
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.217.1
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986536687152-17252-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:16:31.755529 23476 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:16:31.756851 23476 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:16:31.769769 23482 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:16:31.769820 23484 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:16:31.770310 23481 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:16:31.770589 23476 server_base.cc:1034] running on GCE node
I20260620 20:16:31.771284 23476 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:16:31.772189 23476 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:16:31.773432 23476 hybrid_clock.cc:648] HybridClock initialized: now 1781986591773396 us; error 46 us; skew 500 ppm
I20260620 20:16:31.773763 23476 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:16:31 dist-test-slave-2hpg krb5kdc[23370](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986591, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.217.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:16:31.777134 23476 init.cc:377] Logged in from keytab as oryx/127.16.217.1@KRBTEST.COM (short username oryx)
I20260620 20:16:31.778597 23476 webserver.cc:460] Webserver started at http://127.16.217.1:45527/ using document root <none> and password file <none>
I20260620 20:16:31.779217 23476 fs_manager.cc:359] Metadata directory not provided
I20260620 20:16:31.779315 23476 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:16:31.779546 23476 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:16:31.782323 23476 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986536687152-17252-0/minicluster-data/ts-0/data/instance:
uuid: "5280c2f4f3ef4c77890e5b5b7c7f78ae"
format_stamp: "Formatted at 2026-06-20 20:16:31 on dist-test-slave-2hpg"
server_key: "6d099e29af665cf2a57c8e3adfc04bb6"
server_key_iv: "534e6c51d628166c2528284312a65c15"
server_key_version: "encryptionkey@0"
I20260620 20:16:31.783032 23476 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986536687152-17252-0/minicluster-data/ts-0/wal/instance:
uuid: "5280c2f4f3ef4c77890e5b5b7c7f78ae"
format_stamp: "Formatted at 2026-06-20 20:16:31 on dist-test-slave-2hpg"
server_key: "6d099e29af665cf2a57c8e3adfc04bb6"
server_key_iv: "534e6c51d628166c2528284312a65c15"
server_key_version: "encryptionkey@0"
I20260620 20:16:31.787849 23476 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 20:16:31.791064 23491 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:16:31.792362 23476 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 20:16:31.792543 23476 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986536687152-17252-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986536687152-17252-0/minicluster-data/ts-0/wal
uuid: "5280c2f4f3ef4c77890e5b5b7c7f78ae"
format_stamp: "Formatted at 2026-06-20 20:16:31 on dist-test-slave-2hpg"
server_key: "6d099e29af665cf2a57c8e3adfc04bb6"
server_key_iv: "534e6c51d628166c2528284312a65c15"
server_key_version: "encryptionkey@0"
I20260620 20:16:31.792735 23476 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986536687152-17252-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986536687152-17252-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986536687152-17252-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:16:31.810173 23476 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:16:31.814157 23476 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:16:31.814429 23476 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:16:31.815336 23476 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:16:31.816705 23476 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:16:31.816808 23476 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:16:31.816969 23476 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:16:31.817075 23476 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:16:31.849948 23476 rpc_server.cc:307] RPC server started. Bound to: 127.16.217.1:34627
I20260620 20:16:31.850016 23603 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.217.1:34627 every 8 connection(s)
I20260620 20:16:31.851181 23476 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986536687152-17252-0/minicluster-data/ts-0/data/info.pb
I20260620 20:16:31.858255 17252 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu as pid 23476
I20260620 20:16:31.858479 17252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986536687152-17252-0/minicluster-data/ts-0/wal/instance
I20260620 20:16:31.859287 17252 external_mini_cluster.cc:1428] Setting key 4723b403854c76d88f56a410f5ea619c
Jun 20 20:16:31 dist-test-slave-2hpg krb5kdc[23370](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986591, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.217.1@KRBTEST.COM for oryx/127.16.217.62@KRBTEST.COM
I20260620 20:16:31.871450 23606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:31.853575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.217.1:34709 (local address 127.16.217.62:36319)
0620 20:16:31.854028 (+ 453us) server_negotiation.cc:206] Beginning negotiation
0620 20:16:31.854034 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:16:31.855048 (+ 1014us) server_negotiation.cc:407] Connection header received
0620 20:16:31.856731 (+ 1683us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:16:31.856735 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:16:31.856799 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:16:31.856893 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:16:31.858378 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:31.859501 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:31.861223 (+ 1722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:31.861820 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:31.861989 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:31.865175 (+ 3186us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:16:31.865184 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:16:31.865192 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:16:31.865234 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:16:31.868183 (+ 2949us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:16:31.869084 (+ 901us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:16:31.869088 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:16:31.869090 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:16:31.869224 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:16:31.869685 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:16:31.869688 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:16:31.869690 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:16:31.870017 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:16:31.870143 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0620 20:16:31.871012 (+ 869us) server_negotiation.cc:299] Negotiation successful
0620 20:16:31.871182 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":331,"thread_start_us":203,"threads_started":1}
I20260620 20:16:31.872897 23607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:31.854226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:36319 (local address 127.16.217.1:34709)
0620 20:16:31.854895 (+ 669us) negotiation.cc:107] Waiting for socket to connect
0620 20:16:31.854932 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 20:16:31.856488 (+ 1556us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:16:31.857086 (+ 598us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:16:31.857094 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:16:31.857574 (+ 480us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:16:31.858207 (+ 633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:16:31.858231 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:16:31.859653 (+ 1422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:16:31.859657 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:16:31.861091 (+ 1434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:16:31.861102 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:16:31.862039 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:16:31.862044 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:16:31.862243 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:31.862918 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:16:31.862943 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:16:31.864970 (+ 2027us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:16:31.868313 (+ 3343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:16:31.868325 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:16:31.868345 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:16:31.868972 (+ 627us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:16:31.869336 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:16:31.869344 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:16:31.869351 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:16:31.869579 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:16:31.870135 (+ 556us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:16:31.870145 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:16:31.870611 (+ 466us) client_negotiation.cc:769] Sending connection context
0620 20:16:31.870906 (+ 295us) client_negotiation.cc:240] Negotiation successful
0620 20:16:31.871466 (+ 560us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":261,"threads_started":1}
I20260620 20:16:31.874414 23604 heartbeater.cc:347] Connected to a master server at 127.16.217.62:36319
I20260620 20:16:31.874706 23604 heartbeater.cc:464] Registering TS with master...
I20260620 20:16:31.875458 23604 heartbeater.cc:511] Master 127.16.217.62:36319 requested a full tablet report, sending...
I20260620 20:16:31.877700 23415 ts_manager.cc:194] Registered new tserver with Master: 5280c2f4f3ef4c77890e5b5b7c7f78ae (127.16.217.1:34627)
I20260620 20:16:31.879180 23415 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.16.217.1@KRBTEST.COM'} at 127.16.217.1:34709
WARNING: no policy specified for oryx/127.16.217.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.16.217.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.16.217.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986536687152-17252-0/krb5kdc/oryx_127.16.217.2.keytab.
Entry for principal oryx/127.16.217.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986536687152-17252-0/krb5kdc/oryx_127.16.217.2.keytab.
I20260620 20:16:31.906668 17252 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
/tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986536687152-17252-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986536687152-17252-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986536687152-17252-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986536687152-17252-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.217.2:0
--local_ip_for_outbound_sockets=127.16.217.2
--webserver_interface=127.16.217.2
--webserver_port=0
--tserver_master_addrs=127.16.217.62:36319
--keytab_file=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986536687152-17252-0/krb5kdc/oryx_127.16.217.2.keytab
--principal=oryx/127.16.217.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.217.20:38255
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986536687152-17252-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986536687152-17252-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986536687152-17252-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:16:32.227130 23610 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:16:32.227442 23610 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:16:32.227615 23610 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:16:32.234915 23610 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:16:32.235064 23610 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:16:32.235235 23610 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.217.2
I20260620 20:16:32.244920 23610 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.217.20:38255
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986536687152-17252-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986536687152-17252-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986536687152-17252-0/krb5kdc/oryx_127.16.217.2.keytab
--principal=oryx/127.16.217.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.217.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986536687152-17252-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.217.2
--webserver_port=0
--tserver_master_addrs=127.16.217.62:36319
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.217.2
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986536687152-17252-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:16:32.246253 23610 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:16:32.247527 23610 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:16:32.260381 23615 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:16:32.260421 23618 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:16:32.260340 23616 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:16:32.261255 23610 server_base.cc:1034] running on GCE node
I20260620 20:16:32.262009 23610 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:16:32.262992 23610 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:16:32.264236 23610 hybrid_clock.cc:648] HybridClock initialized: now 1781986592264169 us; error 52 us; skew 500 ppm
I20260620 20:16:32.264601 23610 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:16:32 dist-test-slave-2hpg krb5kdc[23370](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986592, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.217.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:16:32.267940 23610 init.cc:377] Logged in from keytab as oryx/127.16.217.2@KRBTEST.COM (short username oryx)
I20260620 20:16:32.269371 23610 webserver.cc:460] Webserver started at http://127.16.217.2:41695/ using document root <none> and password file <none>
I20260620 20:16:32.270119 23610 fs_manager.cc:359] Metadata directory not provided
I20260620 20:16:32.270226 23610 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:16:32.270519 23610 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:16:32.273473 23610 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986536687152-17252-0/minicluster-data/ts-1/data/instance:
uuid: "33fb52c1b00347a8ac5db344e2959cac"
format_stamp: "Formatted at 2026-06-20 20:16:32 on dist-test-slave-2hpg"
server_key: "df26fbfc2e9cd40fc4b1ae167fdaaff2"
server_key_iv: "0d9acd9345547d7dd4b0503c37a03bce"
server_key_version: "encryptionkey@0"
I20260620 20:16:32.274396 23610 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986536687152-17252-0/minicluster-data/ts-1/wal/instance:
uuid: "33fb52c1b00347a8ac5db344e2959cac"
format_stamp: "Formatted at 2026-06-20 20:16:32 on dist-test-slave-2hpg"
server_key: "df26fbfc2e9cd40fc4b1ae167fdaaff2"
server_key_iv: "0d9acd9345547d7dd4b0503c37a03bce"
server_key_version: "encryptionkey@0"
I20260620 20:16:32.279764 23610 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.002s
I20260620 20:16:32.283198 23625 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:16:32.284596 23610 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 20:16:32.284762 23610 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986536687152-17252-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986536687152-17252-0/minicluster-data/ts-1/wal
uuid: "33fb52c1b00347a8ac5db344e2959cac"
format_stamp: "Formatted at 2026-06-20 20:16:32 on dist-test-slave-2hpg"
server_key: "df26fbfc2e9cd40fc4b1ae167fdaaff2"
server_key_iv: "0d9acd9345547d7dd4b0503c37a03bce"
server_key_version: "encryptionkey@0"
I20260620 20:16:32.284955 23610 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986536687152-17252-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986536687152-17252-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986536687152-17252-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:16:32.318011 23610 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:16:32.322081 23610 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:16:32.322369 23610 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:16:32.323310 23610 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:16:32.324615 23610 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:16:32.324684 23610 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:16:32.324834 23610 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:16:32.324883 23610 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:16:32.357255 23610 rpc_server.cc:307] RPC server started. Bound to: 127.16.217.2:41479
I20260620 20:16:32.357343 23737 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.217.2:41479 every 8 connection(s)
I20260620 20:16:32.358568 23610 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986536687152-17252-0/minicluster-data/ts-1/data/info.pb
I20260620 20:16:32.366139 17252 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu as pid 23610
I20260620 20:16:32.366333 17252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986536687152-17252-0/minicluster-data/ts-1/wal/instance
I20260620 20:16:32.366845 17252 external_mini_cluster.cc:1428] Setting key f50cd1d604b6fe25ee9b843c55f085d8
Jun 20 20:16:32 dist-test-slave-2hpg krb5kdc[23370](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986592, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.217.2@KRBTEST.COM for oryx/127.16.217.62@KRBTEST.COM
I20260620 20:16:32.379565 23606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:32.361075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.217.2:48619 (local address 127.16.217.62:36319)
0620 20:16:32.361278 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0620 20:16:32.361284 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:16:32.362518 (+ 1234us) server_negotiation.cc:407] Connection header received
0620 20:16:32.364279 (+ 1761us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:16:32.364283 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:16:32.364345 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:16:32.364430 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:16:32.366126 (+ 1696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:32.367166 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:32.368925 (+ 1759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:32.369606 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:32.369743 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:32.373455 (+ 3712us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:16:32.373464 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:16:32.373472 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:16:32.373514 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:16:32.376451 (+ 2937us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:16:32.377380 (+ 929us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:16:32.377384 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:16:32.377386 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:16:32.377497 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:16:32.377961 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:16:32.377964 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:16:32.377966 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:16:32.378194 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:16:32.378303 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0620 20:16:32.379138 (+ 835us) server_negotiation.cc:299] Negotiation successful
0620 20:16:32.379337 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 20:16:32.381084 23740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:32.361633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:36319 (local address 127.16.217.2:48619)
0620 20:16:32.362358 (+ 725us) negotiation.cc:107] Waiting for socket to connect
0620 20:16:32.362405 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 20:16:32.364023 (+ 1618us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:16:32.364626 (+ 603us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:16:32.364674 (+ 48us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:16:32.365294 (+ 620us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:16:32.365962 (+ 668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:16:32.365982 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:16:32.367305 (+ 1323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:16:32.367308 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:16:32.368795 (+ 1487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:16:32.368806 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:16:32.369927 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:16:32.369936 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:16:32.370157 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:32.370847 (+ 690us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:16:32.370873 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:16:32.373231 (+ 2358us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:16:32.376568 (+ 3337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:16:32.376579 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:16:32.376598 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:16:32.377264 (+ 666us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:16:32.377596 (+ 332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:16:32.377604 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:16:32.377611 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:16:32.377844 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:16:32.378312 (+ 468us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:16:32.378324 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:16:32.378776 (+ 452us) client_negotiation.cc:769] Sending connection context
0620 20:16:32.379063 (+ 287us) client_negotiation.cc:240] Negotiation successful
0620 20:16:32.379608 (+ 545us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":486,"thread_start_us":329,"threads_started":1}
I20260620 20:16:32.382565 23738 heartbeater.cc:347] Connected to a master server at 127.16.217.62:36319
I20260620 20:16:32.382875 23738 heartbeater.cc:464] Registering TS with master...
I20260620 20:16:32.383564 23738 heartbeater.cc:511] Master 127.16.217.62:36319 requested a full tablet report, sending...
I20260620 20:16:32.384980 23415 ts_manager.cc:194] Registered new tserver with Master: 33fb52c1b00347a8ac5db344e2959cac (127.16.217.2:41479)
I20260620 20:16:32.385744 23415 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.16.217.2@KRBTEST.COM'} at 127.16.217.2:48619
WARNING: no policy specified for oryx/127.16.217.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.16.217.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.16.217.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986536687152-17252-0/krb5kdc/oryx_127.16.217.3.keytab.
Entry for principal oryx/127.16.217.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986536687152-17252-0/krb5kdc/oryx_127.16.217.3.keytab.
I20260620 20:16:32.414775 17252 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
/tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986536687152-17252-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986536687152-17252-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986536687152-17252-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986536687152-17252-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.217.3:0
--local_ip_for_outbound_sockets=127.16.217.3
--webserver_interface=127.16.217.3
--webserver_port=0
--tserver_master_addrs=127.16.217.62:36319
--keytab_file=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986536687152-17252-0/krb5kdc/oryx_127.16.217.3.keytab
--principal=oryx/127.16.217.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.217.20:38255
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986536687152-17252-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986536687152-17252-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986536687152-17252-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:16:32.737123 23743 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:16:32.737376 23743 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:16:32.737526 23743 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:16:32.745011 23743 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:16:32.745108 23743 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:16:32.745299 23743 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.217.3
I20260620 20:16:32.754777 23743 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.217.20:38255
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986536687152-17252-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986536687152-17252-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986536687152-17252-0/krb5kdc/oryx_127.16.217.3.keytab
--principal=oryx/127.16.217.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.217.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986536687152-17252-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.217.3
--webserver_port=0
--tserver_master_addrs=127.16.217.62:36319
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.217.3
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986536687152-17252-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:16:32.756000 23743 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:16:32.757323 23743 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:16:32.770221 23748 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:16:32.770377 23749 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:16:32.770370 23751 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:16:32.771332 23743 server_base.cc:1034] running on GCE node
I20260620 20:16:32.771867 23743 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:16:32.772727 23743 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:16:32.773965 23743 hybrid_clock.cc:648] HybridClock initialized: now 1781986592773918 us; error 50 us; skew 500 ppm
I20260620 20:16:32.774283 23743 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:16:32 dist-test-slave-2hpg krb5kdc[23370](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986592, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.217.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:16:32.777498 23743 init.cc:377] Logged in from keytab as oryx/127.16.217.3@KRBTEST.COM (short username oryx)
I20260620 20:16:32.778887 23743 webserver.cc:460] Webserver started at http://127.16.217.3:46693/ using document root <none> and password file <none>
I20260620 20:16:32.779496 23743 fs_manager.cc:359] Metadata directory not provided
I20260620 20:16:32.779592 23743 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:16:32.779834 23743 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:16:32.782691 23743 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986536687152-17252-0/minicluster-data/ts-2/data/instance:
uuid: "b6d268e1e2e944af9e30e4b3e4b3e359"
format_stamp: "Formatted at 2026-06-20 20:16:32 on dist-test-slave-2hpg"
server_key: "a1edf7c2c082c607f2b1516616392cdf"
server_key_iv: "a8d4bd1d1e846917dbb01ca9fb55b819"
server_key_version: "encryptionkey@0"
I20260620 20:16:32.783396 23743 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986536687152-17252-0/minicluster-data/ts-2/wal/instance:
uuid: "b6d268e1e2e944af9e30e4b3e4b3e359"
format_stamp: "Formatted at 2026-06-20 20:16:32 on dist-test-slave-2hpg"
server_key: "a1edf7c2c082c607f2b1516616392cdf"
server_key_iv: "a8d4bd1d1e846917dbb01ca9fb55b819"
server_key_version: "encryptionkey@0"
I20260620 20:16:32.788389 23743 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 20:16:32.791815 23758 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:16:32.793105 23743 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 20:16:32.793226 23743 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986536687152-17252-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986536687152-17252-0/minicluster-data/ts-2/wal
uuid: "b6d268e1e2e944af9e30e4b3e4b3e359"
format_stamp: "Formatted at 2026-06-20 20:16:32 on dist-test-slave-2hpg"
server_key: "a1edf7c2c082c607f2b1516616392cdf"
server_key_iv: "a8d4bd1d1e846917dbb01ca9fb55b819"
server_key_version: "encryptionkey@0"
I20260620 20:16:32.793386 23743 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986536687152-17252-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986536687152-17252-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986536687152-17252-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:16:32.812438 23743 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:16:32.816807 23743 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:16:32.817103 23743 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:16:32.818112 23743 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:16:32.819540 23743 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:16:32.819614 23743 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:16:32.819737 23743 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:16:32.819801 23743 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:16:32.850584 23743 rpc_server.cc:307] RPC server started. Bound to: 127.16.217.3:46165
I20260620 20:16:32.850689 23870 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.217.3:46165 every 8 connection(s)
I20260620 20:16:32.851872 23743 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986536687152-17252-0/minicluster-data/ts-2/data/info.pb
I20260620 20:16:32.854259 17252 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu as pid 23743
I20260620 20:16:32.854439 17252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986536687152-17252-0/minicluster-data/ts-2/wal/instance
I20260620 20:16:32.854967 17252 external_mini_cluster.cc:1428] Setting key 8bc7dde8eaa8ec2dd89b7b4c3c1306f5
Jun 20 20:16:32 dist-test-slave-2hpg krb5kdc[23370](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986592, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.217.3@KRBTEST.COM for oryx/127.16.217.62@KRBTEST.COM
I20260620 20:16:32.872771 23606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:32.854590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.217.3:43525 (local address 127.16.217.62:36319)
0620 20:16:32.854774 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0620 20:16:32.854781 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:16:32.856268 (+ 1487us) server_negotiation.cc:407] Connection header received
0620 20:16:32.858533 (+ 2265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:16:32.858537 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:16:32.858619 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:16:32.858698 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:16:32.860146 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:32.860880 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:32.862600 (+ 1720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:32.862995 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:32.863092 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:32.866641 (+ 3549us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:16:32.866646 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:16:32.866649 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:16:32.866685 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:16:32.869569 (+ 2884us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:16:32.870624 (+ 1055us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:16:32.870630 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:16:32.870634 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:16:32.870828 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:16:32.871264 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:16:32.871267 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:16:32.871269 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:16:32.871541 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:16:32.871651 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0620 20:16:32.872364 (+ 713us) server_negotiation.cc:299] Negotiation successful
0620 20:16:32.872518 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 20:16:32.873778 23873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:32.855258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:36319 (local address 127.16.217.3:43525)
0620 20:16:32.856103 (+ 845us) negotiation.cc:107] Waiting for socket to connect
0620 20:16:32.856161 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0620 20:16:32.858310 (+ 2149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:16:32.858929 (+ 619us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:16:32.858937 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:16:32.859412 (+ 475us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:16:32.859984 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:16:32.860001 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:16:32.861036 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:16:32.861040 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:16:32.862431 (+ 1391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:16:32.862444 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:16:32.863221 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:16:32.863226 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:16:32.863436 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:32.864076 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:16:32.864103 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:16:32.866432 (+ 2329us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:16:32.869741 (+ 3309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:16:32.869751 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:16:32.869771 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:16:32.870492 (+ 721us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:16:32.870953 (+ 461us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:16:32.870957 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:16:32.870959 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:16:32.871142 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:16:32.871646 (+ 504us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:16:32.871652 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:16:32.872018 (+ 366us) client_negotiation.cc:769] Sending connection context
0620 20:16:32.872257 (+ 239us) client_negotiation.cc:240] Negotiation successful
0620 20:16:32.872650 (+ 393us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":582,"thread_start_us":323,"threads_started":1}
I20260620 20:16:32.875221 23871 heartbeater.cc:347] Connected to a master server at 127.16.217.62:36319
I20260620 20:16:32.875480 23871 heartbeater.cc:464] Registering TS with master...
I20260620 20:16:32.876345 23871 heartbeater.cc:511] Master 127.16.217.62:36319 requested a full tablet report, sending...
I20260620 20:16:32.877724 23415 ts_manager.cc:194] Registered new tserver with Master: b6d268e1e2e944af9e30e4b3e4b3e359 (127.16.217.3:46165)
I20260620 20:16:32.878526 23415 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.16.217.3@KRBTEST.COM'} at 127.16.217.3:43525
I20260620 20:16:32.882032 23604 heartbeater.cc:503] Master 127.16.217.62:36319 was elected leader, sending a full tablet report...
I20260620 20:16:32.883394 17252 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.16.217.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.16.217.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.16.217.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986536687152-17252-0/krb5kdc/oryx_127.16.217.61.keytab.
Entry for principal oryx/127.16.217.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986536687152-17252-0/krb5kdc/oryx_127.16.217.61.keytab.
I20260620 20:16:32.929294 17252 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
/tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986536687152-17252-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986536687152-17252-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986536687152-17252-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986536687152-17252-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.217.61:33441
--webserver_interface=127.16.217.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986536687152-17252-0/krb5kdc/oryx_127.16.217.61.keytab
--principal=oryx/127.16.217.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.217.20:38255
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.217.62:36319,127.16.217.61:33441
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986536687152-17252-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986536687152-17252-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986536687152-17252-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:16:33.257920 23876 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:16:33.258188 23876 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:16:33.258275 23876 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:16:33.265501 23876 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:16:33.265612 23876 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:16:33.265658 23876 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:16:33.265713 23876 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:16:33.265756 23876 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:16:33.275869 23876 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.217.20:38255
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986536687152-17252-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986536687152-17252-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.16.217.62:36319,127.16.217.61:33441
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986536687152-17252-0/krb5kdc/oryx_127.16.217.61.keytab
--principal=oryx/127.16.217.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.217.61:33441
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986536687152-17252-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.16.217.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986536687152-17252-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
W20260620 20:16:33.276255 23876 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.16.217.62:36319,127.16.217.61:33441'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260620 20:16:33.277314 23876 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:16:33.278578 23876 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:16:33.290405 23881 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:16:33.290428 23882 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:16:33.290378 23884 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:16:33.291548 23876 server_base.cc:1034] running on GCE node
I20260620 20:16:33.292263 23876 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:16:33.293576 23876 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:16:33.294862 23876 hybrid_clock.cc:648] HybridClock initialized: now 1781986593294800 us; error 36 us; skew 500 ppm
I20260620 20:16:33.295192 23876 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:16:33 dist-test-slave-2hpg krb5kdc[23370](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986593, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.217.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:16:33.298497 23876 init.cc:377] Logged in from keytab as oryx/127.16.217.61@KRBTEST.COM (short username oryx)
I20260620 20:16:33.299949 23876 webserver.cc:460] Webserver started at http://127.16.217.61:39201/ using document root <none> and password file <none>
I20260620 20:16:33.300652 23876 fs_manager.cc:359] Metadata directory not provided
I20260620 20:16:33.300779 23876 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:16:33.301090 23876 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:16:33.304198 23876 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986536687152-17252-0/minicluster-data/master-1/data/instance:
uuid: "0fe54b7d8d1b498f81774ec398b1f6aa"
format_stamp: "Formatted at 2026-06-20 20:16:33 on dist-test-slave-2hpg"
server_key: "48c8de7d672ea9bd2d1f43fdd6d15ac4"
server_key_iv: "e9138480586ea56da5bb7f90d59e5960"
server_key_version: "encryptionkey@0"
I20260620 20:16:33.304965 23876 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986536687152-17252-0/minicluster-data/master-1/wal/instance:
uuid: "0fe54b7d8d1b498f81774ec398b1f6aa"
format_stamp: "Formatted at 2026-06-20 20:16:33 on dist-test-slave-2hpg"
server_key: "48c8de7d672ea9bd2d1f43fdd6d15ac4"
server_key_iv: "e9138480586ea56da5bb7f90d59e5960"
server_key_version: "encryptionkey@0"
I20260620 20:16:33.310012 23876 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260620 20:16:33.313248 23891 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:16:33.314651 23876 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 20:16:33.314832 23876 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986536687152-17252-0/minicluster-data/master-1/data,/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986536687152-17252-0/minicluster-data/master-1/wal
uuid: "0fe54b7d8d1b498f81774ec398b1f6aa"
format_stamp: "Formatted at 2026-06-20 20:16:33 on dist-test-slave-2hpg"
server_key: "48c8de7d672ea9bd2d1f43fdd6d15ac4"
server_key_iv: "e9138480586ea56da5bb7f90d59e5960"
server_key_version: "encryptionkey@0"
I20260620 20:16:33.315054 23876 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986536687152-17252-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986536687152-17252-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986536687152-17252-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:16:33.336862 23876 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:16:33.341071 23876 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:16:33.341346 23876 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:16:33.359786 23876 rpc_server.cc:307] RPC server started. Bound to: 127.16.217.61:33441
I20260620 20:16:33.359872 23943 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.217.61:33441 every 8 connection(s)
I20260620 20:16:33.361425 23876 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986536687152-17252-0/minicluster-data/master-1/data/info.pb
I20260620 20:16:33.365351 23944 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:16:33.368474 17252 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu as pid 23876
I20260620 20:16:33.373160 23944 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.217.62" port: 36319 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 20:16:33 dist-test-slave-2hpg krb5kdc[23370](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986590, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.16.217.61@KRBTEST.COM
Jun 20 20:16:33 dist-test-slave-2hpg krb5kdc[23370](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986593, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.217.61@KRBTEST.COM for oryx/127.16.217.62@KRBTEST.COM
I20260620 20:16:33.388175 23738 heartbeater.cc:503] Master 127.16.217.62:36319 was elected leader, sending a full tablet report...
I20260620 20:16:33.391757 23948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:33.376673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50258 (local address 127.16.217.62:36319)
0620 20:16:33.377177 (+ 504us) server_negotiation.cc:206] Beginning negotiation
0620 20:16:33.377186 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 20:16:33.377778 (+ 592us) server_negotiation.cc:407] Connection header received
0620 20:16:33.377984 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:16:33.377988 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:16:33.378052 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:16:33.378148 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:16:33.379413 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:33.380148 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:33.381608 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:33.382067 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:33.382200 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:33.385621 (+ 3421us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:16:33.385625 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:16:33.385628 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:16:33.385659 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:16:33.388744 (+ 3085us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:16:33.389560 (+ 816us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:16:33.389566 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:16:33.389569 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:16:33.389761 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:16:33.390162 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:16:33.390167 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:16:33.390170 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:16:33.390460 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:16:33.390619 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0620 20:16:33.391135 (+ 516us) server_negotiation.cc:299] Negotiation successful
0620 20:16:33.391349 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":345,"thread_start_us":238,"threads_started":1}
I20260620 20:16:33.392689 23947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:33.370835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59106 (local address 127.16.217.61:33441)
0620 20:16:33.371634 (+ 799us) server_negotiation.cc:206] Beginning negotiation
0620 20:16:33.371649 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 20:16:33.371698 (+ 49us) server_negotiation.cc:407] Connection header received
0620 20:16:33.373807 (+ 2109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:16:33.373836 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:16:33.374393 (+ 557us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:16:33.374820 (+ 427us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:16:33.376584 (+ 1764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:33.378744 (+ 2160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:33.380103 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:33.380799 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:33.380990 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:33.385321 (+ 4331us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:16:33.385329 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:16:33.385340 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:16:33.385368 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:16:33.388274 (+ 2906us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:16:33.389195 (+ 921us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:16:33.389204 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:16:33.389210 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:16:33.389390 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:16:33.390029 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:16:33.390032 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:16:33.390034 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:16:33.390530 (+ 496us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:16:33.390845 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0620 20:16:33.391166 (+ 321us) server_negotiation.cc:299] Negotiation successful
0620 20:16:33.391554 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":29,"server-negotiator.queue_time_us":578,"spinlock_wait_cycles":14080,"thread_start_us":329,"threads_started":1}
I20260620 20:16:33.392707 23949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:33.376846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:36319 (local address 127.0.0.1:50258)
0620 20:16:33.377625 (+ 779us) negotiation.cc:107] Waiting for socket to connect
0620 20:16:33.377667 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 20:16:33.377831 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:16:33.378231 (+ 400us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:16:33.378237 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:16:33.379024 (+ 787us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:16:33.379255 (+ 231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:16:33.379262 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:16:33.380285 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:16:33.380290 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:16:33.381465 (+ 1175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:16:33.381476 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:16:33.382280 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:16:33.382288 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:16:33.382452 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:33.383024 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:16:33.383045 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:16:33.385424 (+ 2379us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:16:33.388872 (+ 3448us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:16:33.388882 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:16:33.388894 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:16:33.389464 (+ 570us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:16:33.389879 (+ 415us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:16:33.389884 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:16:33.389888 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:16:33.390028 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:16:33.390596 (+ 568us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:16:33.390603 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:16:33.390804 (+ 201us) client_negotiation.cc:769] Sending connection context
0620 20:16:33.391051 (+ 247us) client_negotiation.cc:240] Negotiation successful
0620 20:16:33.391748 (+ 697us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":523,"mutex_wait_us":218,"thread_start_us":355,"threads_started":1}
I20260620 20:16:33.394496 23944 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.217.61" port: 33441 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 20:16:33 dist-test-slave-2hpg krb5kdc[23370](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986593, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.217.61@KRBTEST.COM for oryx/127.16.217.61@KRBTEST.COM
I20260620 20:16:33.405988 23949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:33.395160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.61:33441 (local address 127.0.0.1:59118)
0620 20:16:33.395331 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0620 20:16:33.395346 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 20:16:33.395444 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:16:33.395832 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:16:33.395836 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:16:33.396195 (+ 359us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:16:33.396315 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:16:33.396322 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:16:33.397413 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:16:33.397416 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:16:33.398372 (+ 956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:16:33.398389 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:16:33.398993 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:16:33.398997 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:16:33.399126 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:33.399673 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:16:33.399689 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:16:33.401482 (+ 1793us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:16:33.404093 (+ 2611us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:16:33.404097 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:16:33.404099 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:16:33.404440 (+ 341us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:16:33.404725 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:16:33.404728 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:16:33.404730 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:16:33.404843 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:16:33.405274 (+ 431us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:16:33.405277 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:16:33.405398 (+ 121us) client_negotiation.cc:769] Sending connection context
0620 20:16:33.405502 (+ 104us) client_negotiation.cc:240] Negotiation successful
0620 20:16:33.405684 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":64}
I20260620 20:16:33.406176 23947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:33.395375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59118 (local address 127.16.217.61:33441)
0620 20:16:33.395536 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0620 20:16:33.395541 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:16:33.395554 (+ 13us) server_negotiation.cc:407] Connection header received
0620 20:16:33.395594 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:16:33.395600 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:16:33.395662 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:16:33.395809 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:16:33.396445 (+ 636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:33.397287 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:33.398494 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:33.398881 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:33.398974 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:33.401637 (+ 2663us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:16:33.401641 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:16:33.401644 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:16:33.401669 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:16:33.403985 (+ 2316us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:16:33.404532 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:16:33.404535 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:16:33.404538 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:16:33.404637 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:16:33.404928 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:16:33.404931 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:16:33.404933 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:16:33.405181 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:16:33.405286 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0620 20:16:33.405623 (+ 337us) server_negotiation.cc:299] Negotiation successful
0620 20:16:33.405811 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 20:16:33.410599 23944 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0fe54b7d8d1b498f81774ec398b1f6aa: Bootstrap starting.
I20260620 20:16:33.413827 23944 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0fe54b7d8d1b498f81774ec398b1f6aa: Neither blocks nor log segments found. Creating new log.
I20260620 20:16:33.415009 23944 log.cc:824] T 00000000000000000000000000000000 P 0fe54b7d8d1b498f81774ec398b1f6aa: Log is configured to *not* fsync() on all Append() calls
I20260620 20:16:33.417605 23944 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0fe54b7d8d1b498f81774ec398b1f6aa: No bootstrap required, opened a new log
I20260620 20:16:33.424860 23944 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0fe54b7d8d1b498f81774ec398b1f6aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "358753b177ca4441ba3b33b1e5fd7d66" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 36319 } } peers { permanent_uuid: "0fe54b7d8d1b498f81774ec398b1f6aa" member_type: VOTER last_known_addr { host: "127.16.217.61" port: 33441 } }
I20260620 20:16:33.425159 23944 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0fe54b7d8d1b498f81774ec398b1f6aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:16:33.425231 23944 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0fe54b7d8d1b498f81774ec398b1f6aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fe54b7d8d1b498f81774ec398b1f6aa, State: Initialized, Role: FOLLOWER
I20260620 20:16:33.425732 23944 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0fe54b7d8d1b498f81774ec398b1f6aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "358753b177ca4441ba3b33b1e5fd7d66" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 36319 } } peers { permanent_uuid: "0fe54b7d8d1b498f81774ec398b1f6aa" member_type: VOTER last_known_addr { host: "127.16.217.61" port: 33441 } }
I20260620 20:16:33.426554 23951 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0fe54b7d8d1b498f81774ec398b1f6aa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "358753b177ca4441ba3b33b1e5fd7d66" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 36319 } } peers { permanent_uuid: "0fe54b7d8d1b498f81774ec398b1f6aa" member_type: VOTER last_known_addr { host: "127.16.217.61" port: 33441 } } }
I20260620 20:16:33.426844 23951 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0fe54b7d8d1b498f81774ec398b1f6aa [sys.catalog]: This master's current role is: FOLLOWER
I20260620 20:16:33.427582 23944 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0fe54b7d8d1b498f81774ec398b1f6aa [sys.catalog]: configured and running, proceeding with master startup.
W20260620 20:16:33.436215 23962 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0fe54b7d8d1b498f81774ec398b1f6aa: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 20:16:33.436344 23962 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 20:16:33.438462 23876 master_runner.cc:428] Detected that this master 0fe54b7d8d1b498f81774ec398b1f6aa is joining an existing cluster
I20260620 20:16:33.438558 23876 master_runner.cc:432] Initiating AddMaster RPC to add 127.16.217.61:33441
I20260620 20:16:33.459013 23947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:33.442836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59122 (local address 127.16.217.61:33441)
0620 20:16:33.443012 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0620 20:16:33.443018 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:16:33.443507 (+ 489us) server_negotiation.cc:407] Connection header received
0620 20:16:33.443714 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:16:33.443718 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:16:33.443794 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:16:33.443873 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:16:33.445242 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:33.446276 (+ 1034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:33.448273 (+ 1997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:33.448892 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:33.449027 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:33.451600 (+ 2573us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:16:33.451609 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:16:33.451614 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:16:33.451647 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:16:33.455879 (+ 4232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:16:33.456733 (+ 854us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:16:33.456739 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:16:33.456743 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:16:33.456891 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:16:33.457274 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:16:33.457279 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:16:33.457282 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:16:33.457702 (+ 420us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:16:33.457828 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0620 20:16:33.458352 (+ 524us) server_negotiation.cc:299] Negotiation successful
0620 20:16:33.458586 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":18,"server-negotiator.queue_time_us":51,"spinlock_wait_cycles":1792}
I20260620 20:16:33.459369 23948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:33.442453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50272 (local address 127.16.217.62:36319)
0620 20:16:33.442620 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0620 20:16:33.442627 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:16:33.444102 (+ 1475us) server_negotiation.cc:407] Connection header received
0620 20:16:33.444253 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:16:33.444260 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:16:33.444353 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:16:33.444458 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:16:33.445750 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:33.446821 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:33.449463 (+ 2642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:33.450152 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:33.450260 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:33.452696 (+ 2436us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:16:33.452703 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:16:33.452708 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:16:33.452750 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:16:33.456228 (+ 3478us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:16:33.456978 (+ 750us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:16:33.456982 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:16:33.456986 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:16:33.457132 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:16:33.457608 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:16:33.457613 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:16:33.457616 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:16:33.457983 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:16:33.458091 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0620 20:16:33.458651 (+ 560us) server_negotiation.cc:299] Negotiation successful
0620 20:16:33.458867 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 20:16:33.460294 23968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:33.442585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.61:33441 (local address 127.0.0.1:59122)
0620 20:16:33.443385 (+ 800us) negotiation.cc:107] Waiting for socket to connect
0620 20:16:33.443406 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 20:16:33.443554 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:16:33.444036 (+ 482us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:16:33.444038 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:16:33.444762 (+ 724us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:16:33.445107 (+ 345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:16:33.445116 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:16:33.446475 (+ 1359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:16:33.446483 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:16:33.448137 (+ 1654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:16:33.448157 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:16:33.449061 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:16:33.449068 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:16:33.449281 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:33.450000 (+ 719us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:16:33.450024 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:16:33.451407 (+ 1383us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:16:33.456050 (+ 4643us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:16:33.456054 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:16:33.456059 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:16:33.456621 (+ 562us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:16:33.457024 (+ 403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:16:33.457031 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:16:33.457035 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:16:33.457172 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:16:33.457944 (+ 772us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:16:33.457948 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:16:33.458097 (+ 149us) client_negotiation.cc:769] Sending connection context
0620 20:16:33.459477 (+ 1380us) client_negotiation.cc:240] Negotiation successful
0620 20:16:33.459811 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":593,"mutex_wait_us":38,"thread_start_us":459,"threads_started":1}
I20260620 20:16:33.460897 23969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:33.443349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:36319 (local address 127.0.0.1:50272)
0620 20:16:33.443986 (+ 637us) negotiation.cc:107] Waiting for socket to connect
0620 20:16:33.444009 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 20:16:33.444130 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:16:33.444502 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:16:33.444506 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:16:33.445267 (+ 761us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:16:33.445592 (+ 325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:16:33.445604 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:16:33.446988 (+ 1384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:16:33.446993 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:16:33.449308 (+ 2315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:16:33.449320 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:16:33.450546 (+ 1226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:16:33.450552 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:16:33.450727 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:33.451318 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:16:33.451347 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:16:33.452481 (+ 1134us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:16:33.456381 (+ 3900us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:16:33.456387 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:16:33.456390 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:16:33.456851 (+ 461us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:16:33.457280 (+ 429us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:16:33.457284 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:16:33.457287 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:16:33.457483 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:16:33.458171 (+ 688us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:16:33.458175 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:16:33.458387 (+ 212us) client_negotiation.cc:769] Sending connection context
0620 20:16:33.459808 (+ 1421us) client_negotiation.cc:240] Negotiation successful
0620 20:16:33.460128 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":517,"mutex_wait_us":57,"spinlock_wait_cycles":37888,"thread_start_us":327,"threads_started":1}
WARNING: no policy specified for oryx/127.16.217.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.16.217.60@KRBTEST.COM" created.
Jun 20 20:16:33 dist-test-slave-2hpg krb5kdc[23370](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986591, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.217.62@KRBTEST.COM for oryx/127.16.217.61@KRBTEST.COM
I20260620 20:16:33.485451 23947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:33.466812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59130 (local address 127.16.217.61:33441)
0620 20:16:33.466998 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0620 20:16:33.467007 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 20:16:33.470623 (+ 3616us) server_negotiation.cc:407] Connection header received
0620 20:16:33.471009 (+ 386us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:16:33.471015 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:16:33.471122 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:16:33.471223 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:16:33.472469 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:33.473230 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:33.474962 (+ 1732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:33.475645 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:33.475902 (+ 257us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:33.479247 (+ 3345us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:16:33.479259 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:16:33.479263 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:16:33.479294 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:16:33.482699 (+ 3405us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:16:33.483412 (+ 713us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:16:33.483418 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:16:33.483422 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:16:33.483555 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:16:33.483944 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:16:33.483948 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:16:33.483952 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:16:33.484244 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:16:33.484416 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0620 20:16:33.484873 (+ 457us) server_negotiation.cc:299] Negotiation successful
0620 20:16:33.485083 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 20:16:33.485502 23971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:33.466831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.61:33441 (local address 127.0.0.1:59130)
0620 20:16:33.470502 (+ 3671us) negotiation.cc:107] Waiting for socket to connect
0620 20:16:33.470544 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 20:16:33.470753 (+ 209us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:16:33.471294 (+ 541us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:16:33.471303 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:16:33.472036 (+ 733us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:16:33.472312 (+ 276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:16:33.472322 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:16:33.473362 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:16:33.473369 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:16:33.474728 (+ 1359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:16:33.474740 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:16:33.475906 (+ 1166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:16:33.475913 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:16:33.476140 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:33.476892 (+ 752us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:16:33.476910 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:16:33.479042 (+ 2132us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:16:33.482835 (+ 3793us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:16:33.482849 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:16:33.482857 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:16:33.483282 (+ 425us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:16:33.483665 (+ 383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:16:33.483672 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:16:33.483677 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:16:33.483826 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:16:33.484392 (+ 566us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:16:33.484400 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:16:33.484577 (+ 177us) client_negotiation.cc:769] Sending connection context
0620 20:16:33.484763 (+ 186us) client_negotiation.cc:240] Negotiation successful
0620 20:16:33.485083 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3404,"thread_start_us":3231,"threads_started":1}
I20260620 20:16:33.487527 23415 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.16.217.61:33441: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "0fe54b7d8d1b498f81774ec398b1f6aa"
member_type: NON_VOTER
last_known_addr {
host: "127.16.217.61"
port: 33441
}
attrs {
promote: true
}
}
dest_uuid: "358753b177ca4441ba3b33b1e5fd7d66"
cas_config_opid_index: -1
I20260620 20:16:33.488574 23415 consensus_queue.cc:237] T 00000000000000000000000000000000 P 358753b177ca4441ba3b33b1e5fd7d66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "358753b177ca4441ba3b33b1e5fd7d66" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 36319 } } peers { permanent_uuid: "0fe54b7d8d1b498f81774ec398b1f6aa" member_type: NON_VOTER last_known_addr { host: "127.16.217.61" port: 33441 } attrs { promote: true } }
I20260620 20:16:33.499382 23975 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 358753b177ca4441ba3b33b1e5fd7d66 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 0fe54b7d8d1b498f81774ec398b1f6aa (127.16.217.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "358753b177ca4441ba3b33b1e5fd7d66" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 36319 } } peers { permanent_uuid: "0fe54b7d8d1b498f81774ec398b1f6aa" member_type: NON_VOTER last_known_addr { host: "127.16.217.61" port: 33441 } attrs { promote: true } } }
I20260620 20:16:33.502178 23919 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0fe54b7d8d1b498f81774ec398b1f6aa [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:16:33.504035 23975 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.16.217.61:33441
I20260620 20:16:33.504464 23975 sys_catalog.cc:455] T 00000000000000000000000000000000 P 358753b177ca4441ba3b33b1e5fd7d66 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "358753b177ca4441ba3b33b1e5fd7d66" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "358753b177ca4441ba3b33b1e5fd7d66" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 36319 } } peers { permanent_uuid: "0fe54b7d8d1b498f81774ec398b1f6aa" member_type: NON_VOTER last_known_addr { host: "127.16.217.61" port: 33441 } attrs { promote: true } } }
I20260620 20:16:33.504668 23975 sys_catalog.cc:458] T 00000000000000000000000000000000 P 358753b177ca4441ba3b33b1e5fd7d66 [sys.catalog]: This master's current role is: LEADER
I20260620 20:16:33.504942 23876 master.cc:537] Master@127.16.217.61:33441 shutting down...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.16.217.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986536687152-17252-0/krb5kdc/oryx_127.16.217.60.keytab.
Entry for principal oryx/127.16.217.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986536687152-17252-0/krb5kdc/oryx_127.16.217.60.keytab.
I20260620 20:16:33.507180 17252 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
/tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986536687152-17252-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986536687152-17252-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986536687152-17252-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986536687152-17252-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.217.60:37035
--webserver_interface=127.16.217.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986536687152-17252-0/krb5kdc/oryx_127.16.217.60.keytab
--principal=oryx/127.16.217.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.217.20:38255
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.217.62:36319,127.16.217.61:33441,127.16.217.60:37035
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986536687152-17252-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986536687152-17252-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986536687152-17252-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 20:16:33.507833 23919 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 0fe54b7d8d1b498f81774ec398b1f6aa [term 1 FOLLOWER]: Refusing update from remote peer 358753b177ca4441ba3b33b1e5fd7d66: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260620 20:16:33.510167 23974 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 358753b177ca4441ba3b33b1e5fd7d66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0fe54b7d8d1b498f81774ec398b1f6aa" member_type: NON_VOTER last_known_addr { host: "127.16.217.61" port: 33441 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 20:16:33.513513 23400 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
I20260620 20:16:33.519619 23876 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 0fe54b7d8d1b498f81774ec398b1f6aa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 20:16:33.520027 23876 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 0fe54b7d8d1b498f81774ec398b1f6aa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:16:33.520236 23876 tablet_replica.cc:330] T 00000000000000000000000000000000 P 0fe54b7d8d1b498f81774ec398b1f6aa: stopping tablet replica
W20260620 20:16:33.524056 23400 consensus_peers.cc:487] T 00000000000000000000000000000000 P 358753b177ca4441ba3b33b1e5fd7d66 -> Peer 0fe54b7d8d1b498f81774ec398b1f6aa (127.16.217.61:33441): Couldn't send request to peer 0fe54b7d8d1b498f81774ec398b1f6aa. Status: Network error: recv got EOF from unknown peer (error 108). This is attempt 1: this message will repeat every 5th retry.
I20260620 20:16:33.526937 23876 master.cc:552] Master@127.16.217.61:33441 shutdown complete.
I20260620 20:16:33.527026 23876 master_runner.cc:305] Clearing existing system tablet
I20260620 20:16:33.527451 23876 fs_manager.cc:359] Metadata directory not provided
I20260620 20:16:33.527552 23876 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:16:33.530967 23876 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260620 20:16:33.532792 23980 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:16:33.533164 23876 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986536687152-17252-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:16:33.533638 23876 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 20:16:33.533800 23876 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986536687152-17252-0/minicluster-data/master-1/data,/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986536687152-17252-0/minicluster-data/master-1/wal
uuid: "0fe54b7d8d1b498f81774ec398b1f6aa"
format_stamp: "Formatted at 2026-06-20 20:16:33 on dist-test-slave-2hpg"
server_key: "48c8de7d672ea9bd2d1f43fdd6d15ac4"
server_key_iv: "e9138480586ea56da5bb7f90d59e5960"
server_key_version: "encryptionkey@0"
I20260620 20:16:33.540114 23876 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 0fe54b7d8d1b498f81774ec398b1f6aa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:16:33.547950 23876 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 0fe54b7d8d1b498f81774ec398b1f6aa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260620 20:16:33.548211 23876 log.cc:1196] T 00000000000000000000000000000000 P 0fe54b7d8d1b498f81774ec398b1f6aa: Deleting WAL directory at /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986536687152-17252-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260620 20:16:33.548692 23876 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 0fe54b7d8d1b498f81774ec398b1f6aa: Deleting consensus metadata
I20260620 20:16:33.548993 23876 master_runner.cc:315] Copying system tablet from 127.16.217.62:36319
I20260620 20:16:33.551579 23876 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 0fe54b7d8d1b498f81774ec398b1f6aa: tablet copy: Beginning tablet copy session from remote peer at address 127.16.217.62:36319
I20260620 20:16:33.573184 23986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:33.552525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:36319 (local address 127.0.0.1:50282)
0620 20:16:33.558283 (+ 5758us) negotiation.cc:107] Waiting for socket to connect
0620 20:16:33.558309 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0620 20:16:33.558479 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:16:33.558912 (+ 433us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:16:33.558917 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:16:33.559435 (+ 518us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:16:33.559713 (+ 278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:16:33.559726 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:16:33.561206 (+ 1480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:16:33.561212 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:16:33.562725 (+ 1513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:16:33.562741 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:16:33.563755 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:16:33.563761 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:16:33.563930 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:33.564662 (+ 732us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:16:33.564683 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:16:33.565849 (+ 1166us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:16:33.570188 (+ 4339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:16:33.570193 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:16:33.570197 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:16:33.570730 (+ 533us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:16:33.571179 (+ 449us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:16:33.571184 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:16:33.571187 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:16:33.571361 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:16:33.572017 (+ 656us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:16:33.572026 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:16:33.572218 (+ 192us) client_negotiation.cc:769] Sending connection context
0620 20:16:33.572407 (+ 189us) client_negotiation.cc:240] Negotiation successful
0620 20:16:33.572729 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5619,"thread_start_us":5499,"threads_started":1}
I20260620 20:16:33.573300 23948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:33.552548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50282 (local address 127.16.217.62:36319)
0620 20:16:33.552737 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0620 20:16:33.552744 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:16:33.558442 (+ 5698us) server_negotiation.cc:407] Connection header received
0620 20:16:33.558636 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:16:33.558643 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:16:33.558741 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:16:33.558867 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:16:33.559884 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:33.561053 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:33.562890 (+ 1837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:33.563553 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:33.563704 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:33.566105 (+ 2401us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:16:33.566165 (+ 60us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:16:33.566171 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:16:33.566215 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:16:33.570049 (+ 3834us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:16:33.570865 (+ 816us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:16:33.570871 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:16:33.570875 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:16:33.571037 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:16:33.571489 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:16:33.571495 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:16:33.571499 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:16:33.571876 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:16:33.572041 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0620 20:16:33.572647 (+ 606us) server_negotiation.cc:299] Negotiation successful
0620 20:16:33.572874 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 20:16:33.578356 23435 tablet_copy_service.cc:140] P 358753b177ca4441ba3b33b1e5fd7d66: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 0fe54b7d8d1b498f81774ec398b1f6aa ({username='oryx', principal='oryx/127.16.217.61@KRBTEST.COM'} at 127.0.0.1:50282)
I20260620 20:16:33.578696 23435 tablet_copy_service.cc:161] P 358753b177ca4441ba3b33b1e5fd7d66: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 0fe54b7d8d1b498f81774ec398b1f6aa at {username='oryx', principal='oryx/127.16.217.61@KRBTEST.COM'} at 127.0.0.1:50282: session id = 0fe54b7d8d1b498f81774ec398b1f6aa-00000000000000000000000000000000
I20260620 20:16:33.583868 23435 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 358753b177ca4441ba3b33b1e5fd7d66: Tablet Copy: opened 0 blocks and 1 log segments
I20260620 20:16:33.586306 23876 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260620 20:16:33.593240 23876 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 0fe54b7d8d1b498f81774ec398b1f6aa: tablet copy: Starting download of 0 data blocks...
I20260620 20:16:33.593541 23876 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 0fe54b7d8d1b498f81774ec398b1f6aa: tablet copy: Starting download of 1 WAL segments...
I20260620 20:16:33.596449 23876 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 0fe54b7d8d1b498f81774ec398b1f6aa: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260620 20:16:33.601275 23435 tablet_copy_service.cc:342] P 358753b177ca4441ba3b33b1e5fd7d66: Request end of tablet copy session 0fe54b7d8d1b498f81774ec398b1f6aa-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.16.217.61@KRBTEST.COM'} at 127.0.0.1:50282
I20260620 20:16:33.601505 23435 tablet_copy_service.cc:433] P 358753b177ca4441ba3b33b1e5fd7d66: ending tablet copy session 0fe54b7d8d1b498f81774ec398b1f6aa-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 0fe54b7d8d1b498f81774ec398b1f6aa
I20260620 20:16:33.613802 23876 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:16:33.616557 23987 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:16:33.619119 23988 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:16:33.631142 23990 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:16:33.635303 23876 server_base.cc:1034] running on GCE node
I20260620 20:16:33.635811 23876 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:16:33.636535 23876 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:16:33.637835 23876 hybrid_clock.cc:648] HybridClock initialized: now 1781986593637768 us; error 39 us; skew 500 ppm
I20260620 20:16:33.638124 23876 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:16:33 dist-test-slave-2hpg krb5kdc[23370](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986593, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.217.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:16:33.640466 23876 init.cc:377] Logged in from keytab as oryx/127.16.217.61@KRBTEST.COM (short username oryx)
I20260620 20:16:33.642812 23876 webserver.cc:460] Webserver started at http://127.16.217.61:43521/ using document root <none> and password file <none>
I20260620 20:16:33.643210 23876 fs_manager.cc:359] Metadata directory not provided
I20260620 20:16:33.643294 23876 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:16:33.646667 23876 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.001s
I20260620 20:16:33.648727 23997 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:16:33.649209 23876 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20260620 20:16:33.649379 23876 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986536687152-17252-0/minicluster-data/master-1/data,/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986536687152-17252-0/minicluster-data/master-1/wal
uuid: "0fe54b7d8d1b498f81774ec398b1f6aa"
format_stamp: "Formatted at 2026-06-20 20:16:33 on dist-test-slave-2hpg"
server_key: "48c8de7d672ea9bd2d1f43fdd6d15ac4"
server_key_iv: "e9138480586ea56da5bb7f90d59e5960"
server_key_version: "encryptionkey@0"
I20260620 20:16:33.649565 23876 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986536687152-17252-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986536687152-17252-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986536687152-17252-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:16:33.665165 23876 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:16:33.669453 23876 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:16:33.694730 23876 rpc_server.cc:307] RPC server started. Bound to: 127.16.217.61:33441
I20260620 20:16:33.694788 24049 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.217.61:33441 every 8 connection(s)
I20260620 20:16:33.695746 23876 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986536687152-17252-0/minicluster-data/master-1/data/info.pb
I20260620 20:16:33.701488 24050 sys_catalog.cc:263] Verifying existing consensus state
I20260620 20:16:33.705559 24050 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0fe54b7d8d1b498f81774ec398b1f6aa: Bootstrap starting.
I20260620 20:16:33.722787 24050 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0fe54b7d8d1b498f81774ec398b1f6aa: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 20:16:33.723315 24050 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0fe54b7d8d1b498f81774ec398b1f6aa: Bootstrap complete.
I20260620 20:16:33.724349 24050 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0fe54b7d8d1b498f81774ec398b1f6aa [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "358753b177ca4441ba3b33b1e5fd7d66" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 36319 } } peers { permanent_uuid: "0fe54b7d8d1b498f81774ec398b1f6aa" member_type: NON_VOTER last_known_addr { host: "127.16.217.61" port: 33441 } attrs { promote: true } }
I20260620 20:16:33.724603 24050 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0fe54b7d8d1b498f81774ec398b1f6aa [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 0fe54b7d8d1b498f81774ec398b1f6aa, State: Initialized, Role: LEARNER
I20260620 20:16:33.724864 24050 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0fe54b7d8d1b498f81774ec398b1f6aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "358753b177ca4441ba3b33b1e5fd7d66" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 36319 } } peers { permanent_uuid: "0fe54b7d8d1b498f81774ec398b1f6aa" member_type: NON_VOTER last_known_addr { host: "127.16.217.61" port: 33441 } attrs { promote: true } }
I20260620 20:16:33.725663 24053 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0fe54b7d8d1b498f81774ec398b1f6aa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "358753b177ca4441ba3b33b1e5fd7d66" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 36319 } } peers { permanent_uuid: "0fe54b7d8d1b498f81774ec398b1f6aa" member_type: NON_VOTER last_known_addr { host: "127.16.217.61" port: 33441 } attrs { promote: true } } }
I20260620 20:16:33.726052 24053 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0fe54b7d8d1b498f81774ec398b1f6aa [sys.catalog]: This master's current role is: LEARNER
I20260620 20:16:33.726076 24050 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0fe54b7d8d1b498f81774ec398b1f6aa [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:16:33.735347 24064 catalog_manager.cc:1259] Loaded cluster ID: a002c6e7a28743d2b82d56e2d2592ade
I20260620 20:16:33.735452 24064 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 0fe54b7d8d1b498f81774ec398b1f6aa: loading cluster ID for follower catalog manager: success
I20260620 20:16:33.738909 24064 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 0fe54b7d8d1b498f81774ec398b1f6aa: acquiring CA information for follower catalog manager: success
I20260620 20:16:33.741273 24064 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 0fe54b7d8d1b498f81774ec398b1f6aa: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 20:16:33.880924 23871 heartbeater.cc:503] Master 127.16.217.62:36319 was elected leader, sending a full tablet report...
W20260620 20:16:33.905973 23978 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:16:33.906313 23978 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:16:33.906456 23978 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:16:33.914136 23978 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:16:33.914270 23978 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:16:33.914378 23978 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:16:33.914477 23978 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:16:33.914551 23978 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:16:33.924707 23978 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.217.20:38255
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986536687152-17252-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986536687152-17252-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.16.217.62:36319,127.16.217.61:33441,127.16.217.60:37035
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986536687152-17252-0/krb5kdc/oryx_127.16.217.60.keytab
--principal=oryx/127.16.217.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.217.60:37035
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986536687152-17252-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.16.217.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986536687152-17252-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:16:33.926083 23978 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:16:33.927399 23978 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:16:33.939633 24069 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:16:33.939610 24072 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:16:33.939685 24070 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:16:33.940443 23978 server_base.cc:1034] running on GCE node
I20260620 20:16:33.941198 23978 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:16:33.942673 23978 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:16:33.943866 23978 hybrid_clock.cc:648] HybridClock initialized: now 1781986593943823 us; error 32 us; skew 500 ppm
I20260620 20:16:33.944191 23978 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:16:33 dist-test-slave-2hpg krb5kdc[23370](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986593, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.217.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:16:33.947227 23978 init.cc:377] Logged in from keytab as oryx/127.16.217.60@KRBTEST.COM (short username oryx)
I20260620 20:16:33.948975 23978 webserver.cc:460] Webserver started at http://127.16.217.60:45211/ using document root <none> and password file <none>
I20260620 20:16:33.949625 23978 fs_manager.cc:359] Metadata directory not provided
I20260620 20:16:33.949760 23978 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:16:33.950109 23978 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:16:33.953064 23978 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986536687152-17252-0/minicluster-data/master-2/data/instance:
uuid: "5c48720af35c483da74e1243b72c87c3"
format_stamp: "Formatted at 2026-06-20 20:16:33 on dist-test-slave-2hpg"
server_key: "260b8aae21f9e0414f4f057cb096f035"
server_key_iv: "ef3f6485daa0d7d697c707ec073b966f"
server_key_version: "encryptionkey@0"
I20260620 20:16:33.953815 23978 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986536687152-17252-0/minicluster-data/master-2/wal/instance:
uuid: "5c48720af35c483da74e1243b72c87c3"
format_stamp: "Formatted at 2026-06-20 20:16:33 on dist-test-slave-2hpg"
server_key: "260b8aae21f9e0414f4f057cb096f035"
server_key_iv: "ef3f6485daa0d7d697c707ec073b966f"
server_key_version: "encryptionkey@0"
I20260620 20:16:33.958716 23978 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 20:16:33.962014 24079 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:16:33.963528 23978 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 20:16:33.963691 23978 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986536687152-17252-0/minicluster-data/master-2/data,/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986536687152-17252-0/minicluster-data/master-2/wal
uuid: "5c48720af35c483da74e1243b72c87c3"
format_stamp: "Formatted at 2026-06-20 20:16:33 on dist-test-slave-2hpg"
server_key: "260b8aae21f9e0414f4f057cb096f035"
server_key_iv: "ef3f6485daa0d7d697c707ec073b966f"
server_key_version: "encryptionkey@0"
I20260620 20:16:33.963902 23978 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986536687152-17252-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986536687152-17252-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986536687152-17252-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:16:33.987699 23978 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:16:33.991905 23978 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:16:33.992130 23978 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:16:34.011786 23978 rpc_server.cc:307] RPC server started. Bound to: 127.16.217.60:37035
I20260620 20:16:34.011885 24130 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.217.60:37035 every 8 connection(s)
I20260620 20:16:34.013176 23978 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986536687152-17252-0/minicluster-data/master-2/data/info.pb
I20260620 20:16:34.017124 24132 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:16:34.020601 17252 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu as pid 23978
I20260620 20:16:34.025336 24132 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.217.62" port: 36319 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 20:16:34 dist-test-slave-2hpg krb5kdc[23370](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986590, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.16.217.60@KRBTEST.COM
Jun 20 20:16:34 dist-test-slave-2hpg krb5kdc[23370](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986593, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.217.60@KRBTEST.COM for oryx/127.16.217.62@KRBTEST.COM
I20260620 20:16:34.045166 23948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:34.028747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50292 (local address 127.16.217.62:36319)
0620 20:16:34.028964 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0620 20:16:34.028972 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 20:16:34.029743 (+ 771us) server_negotiation.cc:407] Connection header received
0620 20:16:34.030006 (+ 263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:16:34.030010 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:16:34.030069 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:16:34.030147 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:16:34.031908 (+ 1761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:34.032646 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:34.034287 (+ 1641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:34.034976 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:34.035115 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:34.038663 (+ 3548us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:16:34.038670 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:16:34.038675 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:16:34.038761 (+ 86us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:16:34.042055 (+ 3294us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:16:34.042864 (+ 809us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:16:34.042869 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:16:34.042873 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:16:34.043052 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:16:34.043495 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:16:34.043508 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:16:34.043510 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:16:34.043781 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:16:34.043934 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0620 20:16:34.044452 (+ 518us) server_negotiation.cc:299] Negotiation successful
0620 20:16:34.044819 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 20:16:34.046005 24136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:34.028621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:36319 (local address 127.0.0.1:50292)
0620 20:16:34.029586 (+ 965us) negotiation.cc:107] Waiting for socket to connect
0620 20:16:34.029634 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0620 20:16:34.029818 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:16:34.030249 (+ 431us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:16:34.030257 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:16:34.031090 (+ 833us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:16:34.031750 (+ 660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:16:34.031760 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:16:34.032817 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:16:34.032824 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:16:34.034141 (+ 1317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:16:34.034153 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:16:34.035196 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:16:34.035201 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:16:34.035336 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:34.035927 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:16:34.035953 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:16:34.038441 (+ 2488us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:16:34.042190 (+ 3749us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:16:34.042200 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:16:34.042212 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:16:34.042736 (+ 524us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:16:34.043198 (+ 462us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:16:34.043202 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:16:34.043205 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:16:34.043345 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:16:34.043918 (+ 573us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:16:34.043925 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:16:34.044104 (+ 179us) client_negotiation.cc:769] Sending connection context
0620 20:16:34.044355 (+ 251us) client_negotiation.cc:240] Negotiation successful
0620 20:16:34.044959 (+ 604us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":625,"mutex_wait_us":134,"thread_start_us":439,"threads_started":1}
I20260620 20:16:34.046047 24135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:34.022973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48928 (local address 127.16.217.60:37035)
0620 20:16:34.023746 (+ 773us) server_negotiation.cc:206] Beginning negotiation
0620 20:16:34.023761 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 20:16:34.023809 (+ 48us) server_negotiation.cc:407] Connection header received
0620 20:16:34.025898 (+ 2089us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:16:34.025928 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:16:34.026480 (+ 552us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:16:34.026907 (+ 427us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:16:34.028644 (+ 1737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:34.030833 (+ 2189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:34.032151 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:34.032880 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:34.033103 (+ 223us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:34.037631 (+ 4528us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:16:34.037640 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:16:34.037674 (+ 34us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:16:34.037714 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:16:34.041501 (+ 3787us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:16:34.042219 (+ 718us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:16:34.042228 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:16:34.042232 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:16:34.042407 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:16:34.042861 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:16:34.042867 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:16:34.042870 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:16:34.043530 (+ 660us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:16:34.043840 (+ 310us) server_negotiation.cc:1035] Waiting for connection context
0620 20:16:34.044351 (+ 511us) server_negotiation.cc:299] Negotiation successful
0620 20:16:34.044814 (+ 463us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":56,"server-negotiator.queue_time_us":531,"spinlock_wait_cycles":2432,"thread_start_us":359,"threads_started":1}
I20260620 20:16:34.047504 24132 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.217.61" port: 33441 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 20:16:34 dist-test-slave-2hpg krb5kdc[23370](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986593, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.217.60@KRBTEST.COM for oryx/127.16.217.61@KRBTEST.COM
I20260620 20:16:34.060258 24136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:34.048207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.61:33441 (local address 127.0.0.1:59136)
0620 20:16:34.048419 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0620 20:16:34.048437 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 20:16:34.048544 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:16:34.049224 (+ 680us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:16:34.049227 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:16:34.049560 (+ 333us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:16:34.049682 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:16:34.049690 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:16:34.051076 (+ 1386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:16:34.051080 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:16:34.051974 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:16:34.051984 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:16:34.052671 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:16:34.052675 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:16:34.052838 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:34.053394 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:16:34.053409 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:16:34.055235 (+ 1826us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:16:34.057922 (+ 2687us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:16:34.057927 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:16:34.057932 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:16:34.058290 (+ 358us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:16:34.058662 (+ 372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:16:34.058667 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:16:34.058671 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:16:34.058800 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:16:34.059324 (+ 524us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:16:34.059330 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:16:34.059468 (+ 138us) client_negotiation.cc:769] Sending connection context
0620 20:16:34.059606 (+ 138us) client_negotiation.cc:240] Negotiation successful
0620 20:16:34.059915 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":73}
I20260620 20:16:34.060287 24137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:34.048330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59136 (local address 127.16.217.61:33441)
0620 20:16:34.048935 (+ 605us) server_negotiation.cc:206] Beginning negotiation
0620 20:16:34.048944 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 20:16:34.048965 (+ 21us) server_negotiation.cc:407] Connection header received
0620 20:16:34.049031 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:16:34.049035 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:16:34.049097 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:16:34.049197 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:16:34.049831 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:34.050951 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:34.052116 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:34.052554 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:34.052656 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:34.055405 (+ 2749us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:16:34.055410 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:16:34.055413 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:16:34.055441 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:16:34.057759 (+ 2318us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:16:34.058418 (+ 659us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:16:34.058422 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:16:34.058430 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:16:34.058550 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:16:34.058905 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:16:34.058913 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:16:34.058918 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:16:34.059210 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:16:34.059370 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0620 20:16:34.059698 (+ 328us) server_negotiation.cc:299] Negotiation successful
0620 20:16:34.059904 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":462,"thread_start_us":319,"threads_started":1}
I20260620 20:16:34.061477 24132 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.217.60" port: 37035 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 20:16:34 dist-test-slave-2hpg krb5kdc[23370](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986593, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.217.60@KRBTEST.COM for oryx/127.16.217.60@KRBTEST.COM
I20260620 20:16:34.074257 24136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:34.062297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.60:37035 (local address 127.0.0.1:48936)
0620 20:16:34.062492 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0620 20:16:34.062515 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 20:16:34.062660 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:16:34.063006 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:16:34.063010 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:16:34.063399 (+ 389us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:16:34.063544 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:16:34.063553 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:16:34.064560 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:16:34.064566 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:16:34.065470 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:16:34.065481 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:16:34.066284 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:16:34.066290 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:16:34.066559 (+ 269us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:34.067074 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:16:34.067091 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:16:34.069308 (+ 2217us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:16:34.072174 (+ 2866us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:16:34.072177 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:16:34.072180 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:16:34.072510 (+ 330us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:16:34.072831 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:16:34.072834 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:16:34.072837 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:16:34.072970 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:16:34.073412 (+ 442us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:16:34.073416 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:16:34.073529 (+ 113us) client_negotiation.cc:769] Sending connection context
0620 20:16:34.073657 (+ 128us) client_negotiation.cc:240] Negotiation successful
0620 20:16:34.073968 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":71,"mutex_wait_us":25}
I20260620 20:16:34.074338 24135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:34.062272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48936 (local address 127.16.217.60:37035)
0620 20:16:34.062493 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0620 20:16:34.062499 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:16:34.062597 (+ 98us) server_negotiation.cc:407] Connection header received
0620 20:16:34.062820 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:16:34.062826 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:16:34.062890 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:16:34.062992 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:16:34.063680 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:34.064423 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:34.065633 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:34.066132 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:34.066277 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:34.069471 (+ 3194us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:16:34.069476 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:16:34.069479 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:16:34.069506 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:16:34.072058 (+ 2552us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:16:34.072607 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:16:34.072610 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:16:34.072612 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:16:34.072738 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:16:34.073055 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:16:34.073058 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:16:34.073060 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:16:34.073297 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:16:34.073398 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0620 20:16:34.073731 (+ 333us) server_negotiation.cc:299] Negotiation successful
0620 20:16:34.074082 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":225,"server-negotiator.queue_time_us":92}
I20260620 20:16:34.079128 24132 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5c48720af35c483da74e1243b72c87c3: Bootstrap starting.
I20260620 20:16:34.082259 24132 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5c48720af35c483da74e1243b72c87c3: Neither blocks nor log segments found. Creating new log.
I20260620 20:16:34.083256 24132 log.cc:824] T 00000000000000000000000000000000 P 5c48720af35c483da74e1243b72c87c3: Log is configured to *not* fsync() on all Append() calls
I20260620 20:16:34.086103 24132 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5c48720af35c483da74e1243b72c87c3: No bootstrap required, opened a new log
I20260620 20:16:34.093191 24132 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5c48720af35c483da74e1243b72c87c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "358753b177ca4441ba3b33b1e5fd7d66" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 36319 } } peers { permanent_uuid: "0fe54b7d8d1b498f81774ec398b1f6aa" member_type: VOTER last_known_addr { host: "127.16.217.61" port: 33441 } } peers { permanent_uuid: "5c48720af35c483da74e1243b72c87c3" member_type: VOTER last_known_addr { host: "127.16.217.60" port: 37035 } }
I20260620 20:16:34.093516 24132 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5c48720af35c483da74e1243b72c87c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:16:34.093652 24132 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5c48720af35c483da74e1243b72c87c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c48720af35c483da74e1243b72c87c3, State: Initialized, Role: FOLLOWER
I20260620 20:16:34.094321 24132 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5c48720af35c483da74e1243b72c87c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "358753b177ca4441ba3b33b1e5fd7d66" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 36319 } } peers { permanent_uuid: "0fe54b7d8d1b498f81774ec398b1f6aa" member_type: VOTER last_known_addr { host: "127.16.217.61" port: 33441 } } peers { permanent_uuid: "5c48720af35c483da74e1243b72c87c3" member_type: VOTER last_known_addr { host: "127.16.217.60" port: 37035 } }
I20260620 20:16:34.095485 24139 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c48720af35c483da74e1243b72c87c3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "358753b177ca4441ba3b33b1e5fd7d66" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 36319 } } peers { permanent_uuid: "0fe54b7d8d1b498f81774ec398b1f6aa" member_type: VOTER last_known_addr { host: "127.16.217.61" port: 33441 } } peers { permanent_uuid: "5c48720af35c483da74e1243b72c87c3" member_type: VOTER last_known_addr { host: "127.16.217.60" port: 37035 } } }
I20260620 20:16:34.095844 24139 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c48720af35c483da74e1243b72c87c3 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 20:16:34.096545 24132 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5c48720af35c483da74e1243b72c87c3 [sys.catalog]: configured and running, proceeding with master startup.
W20260620 20:16:34.105930 24150 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5c48720af35c483da74e1243b72c87c3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 20:16:34.106091 24150 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 20:16:34.111179 23978 master_runner.cc:428] Detected that this master 5c48720af35c483da74e1243b72c87c3 is joining an existing cluster
I20260620 20:16:34.111325 23978 master_runner.cc:432] Initiating AddMaster RPC to add 127.16.217.60:37035
I20260620 20:16:34.121294 24153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:34.109724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.61:33441 (local address 127.0.0.1:59140)
0620 20:16:34.110254 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0620 20:16:34.110277 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 20:16:34.110395 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:16:34.111247 (+ 852us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:16:34.111252 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:16:34.111287 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:16:34.111509 (+ 222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:16:34.111521 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:16:34.112932 (+ 1411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:16:34.112942 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:16:34.115130 (+ 2188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:16:34.115145 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:16:34.120429 (+ 5284us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:16:34.120436 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:16:34.120630 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:34.120698 (+ 68us) client_negotiation.cc:769] Sending connection context
0620 20:16:34.120806 (+ 108us) client_negotiation.cc:240] Negotiation successful
0620 20:16:34.120953 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":387,"thread_start_us":829,"threads_started":1}
I20260620 20:16:34.122433 24137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:34.110140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59140 (local address 127.16.217.61:33441)
0620 20:16:34.110323 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0620 20:16:34.110329 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:16:34.110349 (+ 20us) server_negotiation.cc:407] Connection header received
0620 20:16:34.110951 (+ 602us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:16:34.110957 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:16:34.111067 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:16:34.111187 (+ 120us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:16:34.111671 (+ 484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:34.112766 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:34.115301 (+ 2535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:34.116602 (+ 1301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:34.116736 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:34.116964 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0620 20:16:34.122051 (+ 5087us) server_negotiation.cc:299] Negotiation successful
0620 20:16:34.122203 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 20:16:34.123620 24025 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 0fe54b7d8d1b498f81774ec398b1f6aa [term 1 LEARNER]: Deduplicated request from leader. Original: 0.0->[1.1-1.5] Dedup: 1.5->[]
I20260620 20:16:34.124394 24053 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0fe54b7d8d1b498f81774ec398b1f6aa [sys.catalog]: SysCatalogTable state changed. Reason: New leader 358753b177ca4441ba3b33b1e5fd7d66. Latest consensus state: current_term: 1 leader_uuid: "358753b177ca4441ba3b33b1e5fd7d66" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "358753b177ca4441ba3b33b1e5fd7d66" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 36319 } } peers { permanent_uuid: "0fe54b7d8d1b498f81774ec398b1f6aa" member_type: NON_VOTER last_known_addr { host: "127.16.217.61" port: 33441 } attrs { promote: true } } }
I20260620 20:16:34.124640 24053 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0fe54b7d8d1b498f81774ec398b1f6aa [sys.catalog]: This master's current role is: LEARNER
I20260620 20:16:34.139798 23948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:34.115827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50298 (local address 127.16.217.62:36319)
0620 20:16:34.115999 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0620 20:16:34.116005 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:16:34.116871 (+ 866us) server_negotiation.cc:407] Connection header received
0620 20:16:34.117058 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:16:34.117063 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:16:34.117155 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:16:34.117251 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:16:34.119461 (+ 2210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:34.120978 (+ 1517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:34.124020 (+ 3042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:34.124908 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:34.125027 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:34.129346 (+ 4319us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:16:34.129356 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:16:34.129364 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:16:34.129418 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:16:34.134058 (+ 4640us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:16:34.136359 (+ 2301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:16:34.136368 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:16:34.136374 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:16:34.136648 (+ 274us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:16:34.137331 (+ 683us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:16:34.137339 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:16:34.137345 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:16:34.137903 (+ 558us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:16:34.138077 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0620 20:16:34.138931 (+ 854us) server_negotiation.cc:299] Negotiation successful
0620 20:16:34.139240 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 20:16:34.141678 24158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:34.116167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:36319 (local address 127.0.0.1:50298)
0620 20:16:34.116760 (+ 593us) negotiation.cc:107] Waiting for socket to connect
0620 20:16:34.116780 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 20:16:34.116936 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:16:34.117838 (+ 902us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:16:34.117843 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:16:34.118780 (+ 937us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:16:34.119250 (+ 470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:16:34.119267 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:16:34.121229 (+ 1962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:16:34.121237 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:16:34.123801 (+ 2564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:16:34.123823 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:16:34.125351 (+ 1528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:16:34.125356 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:16:34.125569 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:34.126872 (+ 1303us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:16:34.126905 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:16:34.129101 (+ 2196us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:16:34.135502 (+ 6401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:16:34.135510 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:16:34.135517 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:16:34.136173 (+ 656us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:16:34.136923 (+ 750us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:16:34.136930 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:16:34.136937 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:16:34.137178 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:16:34.138176 (+ 998us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:16:34.138188 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:16:34.138502 (+ 314us) client_negotiation.cc:769] Sending connection context
0620 20:16:34.140127 (+ 1625us) client_negotiation.cc:240] Negotiation successful
0620 20:16:34.140590 (+ 463us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":461,"thread_start_us":332,"threads_started":1}
I20260620 20:16:34.150982 24137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:34.117839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59148 (local address 127.16.217.61:33441)
0620 20:16:34.122664 (+ 4825us) server_negotiation.cc:206] Beginning negotiation
0620 20:16:34.122670 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:16:34.126647 (+ 3977us) server_negotiation.cc:407] Connection header received
0620 20:16:34.126696 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:16:34.126701 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:16:34.126784 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:16:34.126868 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:16:34.130185 (+ 3317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:34.131271 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:34.135027 (+ 3756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:34.135731 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:34.135833 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:34.140603 (+ 4770us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:16:34.140611 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:16:34.140616 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:16:34.140664 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:16:34.145304 (+ 4640us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:16:34.146217 (+ 913us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:16:34.146223 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:16:34.146226 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:16:34.146380 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:16:34.147576 (+ 1196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:16:34.147581 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:16:34.147585 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:16:34.147965 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:16:34.148120 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0620 20:16:34.150035 (+ 1915us) server_negotiation.cc:299] Negotiation successful
0620 20:16:34.150441 (+ 406us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4715,"threads_started":1}
I20260620 20:16:34.152484 24159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:34.117399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.61:33441 (local address 127.0.0.1:59148)
0620 20:16:34.126415 (+ 9016us) negotiation.cc:107] Waiting for socket to connect
0620 20:16:34.126439 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 20:16:34.126551 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:16:34.127013 (+ 462us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:16:34.127018 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:16:34.127744 (+ 726us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:16:34.128508 (+ 764us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:16:34.128521 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:16:34.131489 (+ 2968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:16:34.131494 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:16:34.133090 (+ 1596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:16:34.133103 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:16:34.136148 (+ 3045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:16:34.136159 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:16:34.136357 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:34.137591 (+ 1234us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:16:34.137613 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:16:34.138754 (+ 1141us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:16:34.145513 (+ 6759us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:16:34.145519 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:16:34.145524 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:16:34.146068 (+ 544us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:16:34.147234 (+ 1166us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:16:34.147239 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:16:34.147243 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:16:34.147432 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:16:34.148514 (+ 1082us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:16:34.148520 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:16:34.149343 (+ 823us) client_negotiation.cc:769] Sending connection context
0620 20:16:34.149501 (+ 158us) client_negotiation.cc:240] Negotiation successful
0620 20:16:34.149839 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8864,"thread_start_us":8450,"threads_started":1}
I20260620 20:16:34.152987 17252 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu pid 23476
I20260620 20:16:34.168884 24135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:34.117948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48944 (local address 127.16.217.60:37035)
0620 20:16:34.118129 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0620 20:16:34.118135 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:16:34.152755 (+ 34620us) server_negotiation.cc:407] Connection header received
0620 20:16:34.153272 (+ 517us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:16:34.153277 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:16:34.153382 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:16:34.153455 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:16:34.154769 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:34.155868 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:34.157816 (+ 1948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:34.158595 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:34.158688 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:34.161494 (+ 2806us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:16:34.161500 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:16:34.161505 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:16:34.161537 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:16:34.165338 (+ 3801us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:16:34.166294 (+ 956us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:16:34.166299 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:16:34.166303 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:16:34.166423 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:16:34.167206 (+ 783us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:16:34.167212 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:16:34.167219 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:16:34.167612 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:16:34.167738 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0620 20:16:34.168308 (+ 570us) server_negotiation.cc:299] Negotiation successful
0620 20:16:34.168517 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 20:16:34.176144 24161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:34.118098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.60:37035 (local address 127.0.0.1:48944)
0620 20:16:34.152671 (+ 34573us) negotiation.cc:107] Waiting for socket to connect
0620 20:16:34.152689 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 20:16:34.153052 (+ 363us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:16:34.153641 (+ 589us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:16:34.153645 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:16:34.154349 (+ 704us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:16:34.154621 (+ 272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:16:34.154633 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:16:34.156083 (+ 1450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:16:34.156088 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:16:34.157675 (+ 1587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:16:34.157689 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:16:34.158974 (+ 1285us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:16:34.158982 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:16:34.159135 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:34.160225 (+ 1090us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:16:34.160247 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:16:34.161308 (+ 1061us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:16:34.165567 (+ 4259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:16:34.165572 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:16:34.165576 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:16:34.166162 (+ 586us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:16:34.166936 (+ 774us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:16:34.166942 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:16:34.166945 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:16:34.167091 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:16:34.167799 (+ 708us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:16:34.167803 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:16:34.167968 (+ 165us) client_negotiation.cc:769] Sending connection context
0620 20:16:34.175349 (+ 7381us) client_negotiation.cc:240] Negotiation successful
0620 20:16:34.175654 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34307,"thread_start_us":34673,"threads_started":1}
Jun 20 20:16:34 dist-test-slave-2hpg krb5kdc[23370](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986591, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.217.62@KRBTEST.COM for oryx/127.16.217.60@KRBTEST.COM
Jun 20 20:16:34 dist-test-slave-2hpg krb5kdc[23370](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986590, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.16.217.1@KRBTEST.COM
I20260620 20:16:34.186004 24153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:34.148526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.60:37035 (local address 127.0.0.1:48946)
0620 20:16:34.148766 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0620 20:16:34.148786 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 20:16:34.148887 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:16:34.169687 (+ 20800us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:16:34.169695 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:16:34.170214 (+ 519us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:16:34.170443 (+ 229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:16:34.170456 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:16:34.172816 (+ 2360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:16:34.172822 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:16:34.174248 (+ 1426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:16:34.174263 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:16:34.175233 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:16:34.175240 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:16:34.175420 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:34.176238 (+ 818us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:16:34.176265 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:16:34.179524 (+ 3259us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:16:34.183330 (+ 3806us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:16:34.183337 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:16:34.183341 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:16:34.183711 (+ 370us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:16:34.184109 (+ 398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:16:34.184115 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:16:34.184119 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:16:34.184278 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:16:34.184928 (+ 650us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:16:34.184935 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:16:34.185128 (+ 193us) client_negotiation.cc:769] Sending connection context
0620 20:16:34.185296 (+ 168us) client_negotiation.cc:240] Negotiation successful
0620 20:16:34.185581 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52}
I20260620 20:16:34.186048 24135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:34.149054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48946 (local address 127.16.217.60:37035)
0620 20:16:34.169172 (+ 20118us) server_negotiation.cc:206] Beginning negotiation
0620 20:16:34.169177 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:16:34.169191 (+ 14us) server_negotiation.cc:407] Connection header received
0620 20:16:34.169323 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:16:34.169330 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:16:34.169493 (+ 163us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:16:34.171061 (+ 1568us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:16:34.171257 (+ 196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:34.172653 (+ 1396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:34.174429 (+ 1776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:34.175091 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:34.175222 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:34.179731 (+ 4509us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:16:34.179739 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:16:34.179744 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:16:34.179777 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:16:34.183196 (+ 3419us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:16:34.183851 (+ 655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:16:34.183858 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:16:34.183867 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:16:34.183999 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:16:34.184417 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:16:34.184427 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:16:34.184432 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:16:34.184783 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:16:34.184926 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0620 20:16:34.185441 (+ 515us) server_negotiation.cc:299] Negotiation successful
0620 20:16:34.185656 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20032,"thread_start_us":24304,"threads_started":1}
I20260620 20:16:34.187570 23415 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.16.217.60:37035: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "5c48720af35c483da74e1243b72c87c3"
member_type: NON_VOTER
last_known_addr {
host: "127.16.217.60"
port: 37035
}
attrs {
promote: true
}
}
dest_uuid: "358753b177ca4441ba3b33b1e5fd7d66"
cas_config_opid_index: 5
I20260620 20:16:34.188591 23415 consensus_queue.cc:237] T 00000000000000000000000000000000 P 358753b177ca4441ba3b33b1e5fd7d66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "358753b177ca4441ba3b33b1e5fd7d66" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 36319 } } peers { permanent_uuid: "0fe54b7d8d1b498f81774ec398b1f6aa" member_type: NON_VOTER last_known_addr { host: "127.16.217.61" port: 33441 } attrs { promote: true } } peers { permanent_uuid: "5c48720af35c483da74e1243b72c87c3" member_type: NON_VOTER last_known_addr { host: "127.16.217.60" port: 37035 } attrs { promote: true } }
I20260620 20:16:34.189474 24163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:34.154161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35952 (local address 127.16.217.1:34627)
0620 20:16:34.166830 (+ 12669us) server_negotiation.cc:206] Beginning negotiation
0620 20:16:34.166839 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 20:16:34.166870 (+ 31us) server_negotiation.cc:407] Connection header received
0620 20:16:34.166937 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:16:34.166947 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:16:34.167267 (+ 320us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:16:34.170318 (+ 3051us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:16:34.170640 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:34.173047 (+ 2407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:34.174938 (+ 1891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:34.175972 (+ 1034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:34.178488 (+ 2516us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:34.181896 (+ 3408us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:16:34.181913 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:16:34.181933 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:16:34.181983 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:16:34.185642 (+ 3659us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:16:34.186934 (+ 1292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:16:34.186943 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:16:34.186946 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:16:34.187098 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:16:34.187551 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:16:34.187556 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:16:34.187559 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:16:34.188033 (+ 474us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:16:34.188326 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0620 20:16:34.188687 (+ 361us) server_negotiation.cc:299] Negotiation successful
0620 20:16:34.189005 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12490,"thread_start_us":12920,"threads_started":1}
I20260620 20:16:34.191036 24025 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 0fe54b7d8d1b498f81774ec398b1f6aa [term 1 LEARNER]: Refusing update from remote peer 358753b177ca4441ba3b33b1e5fd7d66: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260620 20:16:34.191095 23599 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:16:34.191583 24152 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 358753b177ca4441ba3b33b1e5fd7d66 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 5c48720af35c483da74e1243b72c87c3 (127.16.217.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "358753b177ca4441ba3b33b1e5fd7d66" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 36319 } } peers { permanent_uuid: "0fe54b7d8d1b498f81774ec398b1f6aa" member_type: NON_VOTER last_known_addr { host: "127.16.217.61" port: 33441 } attrs { promote: true } } peers { permanent_uuid: "5c48720af35c483da74e1243b72c87c3" member_type: NON_VOTER last_known_addr { host: "127.16.217.60" port: 37035 } attrs { promote: true } } }
I20260620 20:16:34.194267 24164 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 358753b177ca4441ba3b33b1e5fd7d66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0fe54b7d8d1b498f81774ec398b1f6aa" member_type: NON_VOTER last_known_addr { host: "127.16.217.61" port: 33441 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260620 20:16:34.195695 24152 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.16.217.60:37035
I20260620 20:16:34.196628 23978 master.cc:537] Master@127.16.217.60:37035 shutting down...
I20260620 20:16:34.198468 24164 sys_catalog.cc:455] T 00000000000000000000000000000000 P 358753b177ca4441ba3b33b1e5fd7d66 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "358753b177ca4441ba3b33b1e5fd7d66" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "358753b177ca4441ba3b33b1e5fd7d66" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 36319 } } peers { permanent_uuid: "0fe54b7d8d1b498f81774ec398b1f6aa" member_type: NON_VOTER last_known_addr { host: "127.16.217.61" port: 33441 } attrs { promote: true } } peers { permanent_uuid: "5c48720af35c483da74e1243b72c87c3" member_type: NON_VOTER last_known_addr { host: "127.16.217.60" port: 37035 } attrs { promote: true } } }
I20260620 20:16:34.198979 24164 sys_catalog.cc:458] T 00000000000000000000000000000000 P 358753b177ca4441ba3b33b1e5fd7d66 [sys.catalog]: This master's current role is: LEADER
I20260620 20:16:34.204476 24025 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 0fe54b7d8d1b498f81774ec398b1f6aa [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 5c48720af35c483da74e1243b72c87c3 (127.16.217.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "358753b177ca4441ba3b33b1e5fd7d66" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 36319 } } peers { permanent_uuid: "0fe54b7d8d1b498f81774ec398b1f6aa" member_type: NON_VOTER last_known_addr { host: "127.16.217.61" port: 33441 } attrs { promote: true } } peers { permanent_uuid: "5c48720af35c483da74e1243b72c87c3" member_type: NON_VOTER last_known_addr { host: "127.16.217.60" port: 37035 } attrs { promote: true } } }
I20260620 20:16:34.212142 24053 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0fe54b7d8d1b498f81774ec398b1f6aa [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "358753b177ca4441ba3b33b1e5fd7d66" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "358753b177ca4441ba3b33b1e5fd7d66" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 36319 } } peers { permanent_uuid: "0fe54b7d8d1b498f81774ec398b1f6aa" member_type: NON_VOTER last_known_addr { host: "127.16.217.61" port: 33441 } attrs { promote: true } } peers { permanent_uuid: "5c48720af35c483da74e1243b72c87c3" member_type: NON_VOTER last_known_addr { host: "127.16.217.60" port: 37035 } attrs { promote: true } } }
I20260620 20:16:34.212438 24053 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0fe54b7d8d1b498f81774ec398b1f6aa [sys.catalog]: This master's current role is: LEARNER
I20260620 20:16:34.212836 24152 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 358753b177ca4441ba3b33b1e5fd7d66: attempting to promote NON_VOTER 0fe54b7d8d1b498f81774ec398b1f6aa to VOTER
I20260620 20:16:34.213523 23978 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 5c48720af35c483da74e1243b72c87c3 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260620 20:16:34.214007 23978 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 5c48720af35c483da74e1243b72c87c3 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260620 20:16:34.213902 24152 consensus_queue.cc:237] T 00000000000000000000000000000000 P 358753b177ca4441ba3b33b1e5fd7d66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "358753b177ca4441ba3b33b1e5fd7d66" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 36319 } } peers { permanent_uuid: "0fe54b7d8d1b498f81774ec398b1f6aa" member_type: VOTER last_known_addr { host: "127.16.217.61" port: 33441 } attrs { promote: false } } peers { permanent_uuid: "5c48720af35c483da74e1243b72c87c3" member_type: NON_VOTER last_known_addr { host: "127.16.217.60" port: 37035 } attrs { promote: true } }
I20260620 20:16:34.214349 23978 tablet_replica.cc:330] T 00000000000000000000000000000000 P 5c48720af35c483da74e1243b72c87c3: stopping tablet replica
W20260620 20:16:34.218072 23401 consensus_peers.cc:487] T 00000000000000000000000000000000 P 358753b177ca4441ba3b33b1e5fd7d66 -> Peer 5c48720af35c483da74e1243b72c87c3 (127.16.217.60:37035): Couldn't send request to peer 5c48720af35c483da74e1243b72c87c3. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260620 20:16:34.221306 23978 master.cc:552] Master@127.16.217.60:37035 shutdown complete.
I20260620 20:16:34.221412 23978 master_runner.cc:305] Clearing existing system tablet
I20260620 20:16:34.221784 23978 fs_manager.cc:359] Metadata directory not provided
I20260620 20:16:34.221930 23978 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:16:34.222221 24025 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 0fe54b7d8d1b498f81774ec398b1f6aa [term 1 LEARNER]: Refusing update from remote peer 358753b177ca4441ba3b33b1e5fd7d66: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260620 20:16:34.222766 24169 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 358753b177ca4441ba3b33b1e5fd7d66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0fe54b7d8d1b498f81774ec398b1f6aa" member_type: VOTER last_known_addr { host: "127.16.217.61" port: 33441 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260620 20:16:34.225242 24152 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 358753b177ca4441ba3b33b1e5fd7d66 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 0fe54b7d8d1b498f81774ec398b1f6aa (127.16.217.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "358753b177ca4441ba3b33b1e5fd7d66" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 36319 } } peers { permanent_uuid: "0fe54b7d8d1b498f81774ec398b1f6aa" member_type: VOTER last_known_addr { host: "127.16.217.61" port: 33441 } attrs { promote: false } } peers { permanent_uuid: "5c48720af35c483da74e1243b72c87c3" member_type: NON_VOTER last_known_addr { host: "127.16.217.60" port: 37035 } attrs { promote: true } } }
I20260620 20:16:34.230227 24024 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 0fe54b7d8d1b498f81774ec398b1f6aa [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 0fe54b7d8d1b498f81774ec398b1f6aa (127.16.217.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "358753b177ca4441ba3b33b1e5fd7d66" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 36319 } } peers { permanent_uuid: "0fe54b7d8d1b498f81774ec398b1f6aa" member_type: VOTER last_known_addr { host: "127.16.217.61" port: 33441 } attrs { promote: false } } peers { permanent_uuid: "5c48720af35c483da74e1243b72c87c3" member_type: NON_VOTER last_known_addr { host: "127.16.217.60" port: 37035 } attrs { promote: true } } }
I20260620 20:16:34.234543 24053 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0fe54b7d8d1b498f81774ec398b1f6aa [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "358753b177ca4441ba3b33b1e5fd7d66" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "358753b177ca4441ba3b33b1e5fd7d66" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 36319 } } peers { permanent_uuid: "0fe54b7d8d1b498f81774ec398b1f6aa" member_type: VOTER last_known_addr { host: "127.16.217.61" port: 33441 } attrs { promote: false } } peers { permanent_uuid: "5c48720af35c483da74e1243b72c87c3" member_type: NON_VOTER last_known_addr { host: "127.16.217.60" port: 37035 } attrs { promote: true } } }
I20260620 20:16:34.234808 24053 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0fe54b7d8d1b498f81774ec398b1f6aa [sys.catalog]: This master's current role is: FOLLOWER
I20260620 20:16:34.235564 24152 sys_catalog.cc:455] T 00000000000000000000000000000000 P 358753b177ca4441ba3b33b1e5fd7d66 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "358753b177ca4441ba3b33b1e5fd7d66" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "358753b177ca4441ba3b33b1e5fd7d66" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 36319 } } peers { permanent_uuid: "0fe54b7d8d1b498f81774ec398b1f6aa" member_type: VOTER last_known_addr { host: "127.16.217.61" port: 33441 } attrs { promote: false } } peers { permanent_uuid: "5c48720af35c483da74e1243b72c87c3" member_type: NON_VOTER last_known_addr { host: "127.16.217.60" port: 37035 } attrs { promote: true } } }
I20260620 20:16:34.235618 24169 sys_catalog.cc:455] T 00000000000000000000000000000000 P 358753b177ca4441ba3b33b1e5fd7d66 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "358753b177ca4441ba3b33b1e5fd7d66" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "358753b177ca4441ba3b33b1e5fd7d66" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 36319 } } peers { permanent_uuid: "0fe54b7d8d1b498f81774ec398b1f6aa" member_type: VOTER last_known_addr { host: "127.16.217.61" port: 33441 } attrs { promote: false } } peers { permanent_uuid: "5c48720af35c483da74e1243b72c87c3" member_type: NON_VOTER last_known_addr { host: "127.16.217.60" port: 37035 } attrs { promote: true } } }
I20260620 20:16:34.235859 24152 sys_catalog.cc:458] T 00000000000000000000000000000000 P 358753b177ca4441ba3b33b1e5fd7d66 [sys.catalog]: This master's current role is: LEADER
I20260620 20:16:34.235942 24169 sys_catalog.cc:458] T 00000000000000000000000000000000 P 358753b177ca4441ba3b33b1e5fd7d66 [sys.catalog]: This master's current role is: LEADER
I20260620 20:16:34.242597 23978 fs_manager.cc:709] Time spent opening directory manager: real 0.018s user 0.005s sys 0.011s
I20260620 20:16:34.254758 24172 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:16:34.255151 23978 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986536687152-17252-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:16:34.255453 23978 fs_manager.cc:725] Time spent opening block manager: real 0.012s user 0.005s sys 0.005s
I20260620 20:16:34.255553 23978 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986536687152-17252-0/minicluster-data/master-2/data,/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986536687152-17252-0/minicluster-data/master-2/wal
uuid: "5c48720af35c483da74e1243b72c87c3"
format_stamp: "Formatted at 2026-06-20 20:16:33 on dist-test-slave-2hpg"
server_key: "260b8aae21f9e0414f4f057cb096f035"
server_key_iv: "ef3f6485daa0d7d697c707ec073b966f"
server_key_version: "encryptionkey@0"
I20260620 20:16:34.259132 23978 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 5c48720af35c483da74e1243b72c87c3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:16:34.263792 23978 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 5c48720af35c483da74e1243b72c87c3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260620 20:16:34.263965 23978 log.cc:1196] T 00000000000000000000000000000000 P 5c48720af35c483da74e1243b72c87c3: Deleting WAL directory at /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986536687152-17252-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260620 20:16:34.264369 23978 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 5c48720af35c483da74e1243b72c87c3: Deleting consensus metadata
I20260620 20:16:34.264607 23978 master_runner.cc:315] Copying system tablet from 127.16.217.62:36319
I20260620 20:16:34.267246 23978 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 5c48720af35c483da74e1243b72c87c3: tablet copy: Beginning tablet copy session from remote peer at address 127.16.217.62:36319
I20260620 20:16:34.280367 23948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:34.268027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50314 (local address 127.16.217.62:36319)
0620 20:16:34.268369 (+ 342us) server_negotiation.cc:206] Beginning negotiation
0620 20:16:34.268375 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:16:34.268647 (+ 272us) server_negotiation.cc:407] Connection header received
0620 20:16:34.268869 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:16:34.268875 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:16:34.268949 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:16:34.269055 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:16:34.269929 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:34.270800 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:34.272067 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:34.272478 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:34.272612 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:34.274829 (+ 2217us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:16:34.274843 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:16:34.274847 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:16:34.274878 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:16:34.277844 (+ 2966us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:16:34.278502 (+ 658us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:16:34.278509 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:16:34.278513 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:16:34.278649 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:16:34.279020 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:16:34.279028 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:16:34.279032 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:16:34.279299 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:16:34.279454 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0620 20:16:34.279812 (+ 358us) server_negotiation.cc:299] Negotiation successful
0620 20:16:34.280014 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":181}
I20260620 20:16:34.280454 24178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:34.268028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:36319 (local address 127.0.0.1:50314)
0620 20:16:34.268525 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0620 20:16:34.268545 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 20:16:34.268726 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:16:34.269076 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:16:34.269081 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:16:34.269570 (+ 489us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:16:34.269751 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:16:34.269760 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:16:34.270944 (+ 1184us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:16:34.270950 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:16:34.271923 (+ 973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:16:34.271934 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:16:34.272630 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:16:34.272636 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:16:34.272818 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:34.273451 (+ 633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:16:34.273468 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:16:34.274622 (+ 1154us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:16:34.278003 (+ 3381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:16:34.278009 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:16:34.278013 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:16:34.278384 (+ 371us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:16:34.278767 (+ 383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:16:34.278772 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:16:34.278776 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:16:34.278915 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:16:34.279415 (+ 500us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:16:34.279421 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:16:34.279581 (+ 160us) client_negotiation.cc:769] Sending connection context
0620 20:16:34.279719 (+ 138us) client_negotiation.cc:240] Negotiation successful
0620 20:16:34.280023 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":347,"threads_started":1}
I20260620 20:16:34.281391 23435 tablet_copy_service.cc:140] P 358753b177ca4441ba3b33b1e5fd7d66: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 5c48720af35c483da74e1243b72c87c3 ({username='oryx', principal='oryx/127.16.217.60@KRBTEST.COM'} at 127.0.0.1:50314)
I20260620 20:16:34.281574 23435 tablet_copy_service.cc:161] P 358753b177ca4441ba3b33b1e5fd7d66: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 5c48720af35c483da74e1243b72c87c3 at {username='oryx', principal='oryx/127.16.217.60@KRBTEST.COM'} at 127.0.0.1:50314: session id = 5c48720af35c483da74e1243b72c87c3-00000000000000000000000000000000
I20260620 20:16:34.283815 23435 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 358753b177ca4441ba3b33b1e5fd7d66: Tablet Copy: opened 0 blocks and 1 log segments
I20260620 20:16:34.285368 23978 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260620 20:16:34.290735 23978 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 5c48720af35c483da74e1243b72c87c3: tablet copy: Starting download of 0 data blocks...
I20260620 20:16:34.291016 23978 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 5c48720af35c483da74e1243b72c87c3: tablet copy: Starting download of 1 WAL segments...
I20260620 20:16:34.293054 23978 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 5c48720af35c483da74e1243b72c87c3: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260620 20:16:34.296243 23435 tablet_copy_service.cc:342] P 358753b177ca4441ba3b33b1e5fd7d66: Request end of tablet copy session 5c48720af35c483da74e1243b72c87c3-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.16.217.60@KRBTEST.COM'} at 127.0.0.1:50314
I20260620 20:16:34.296383 23435 tablet_copy_service.cc:433] P 358753b177ca4441ba3b33b1e5fd7d66: ending tablet copy session 5c48720af35c483da74e1243b72c87c3-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 5c48720af35c483da74e1243b72c87c3
I20260620 20:16:34.308596 23978 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:16:34.311362 24179 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:16:34.312463 24180 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:16:34.319360 24182 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:16:34.320165 23978 server_base.cc:1034] running on GCE node
I20260620 20:16:34.320515 23978 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:16:34.321206 23978 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:16:34.322402 23978 hybrid_clock.cc:648] HybridClock initialized: now 1781986594322353 us; error 42 us; skew 500 ppm
I20260620 20:16:34.322664 23978 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:16:34 dist-test-slave-2hpg krb5kdc[23370](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986594, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.217.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:16:34.324796 23978 init.cc:377] Logged in from keytab as oryx/127.16.217.60@KRBTEST.COM (short username oryx)
I20260620 20:16:34.326113 23978 webserver.cc:460] Webserver started at http://127.16.217.60:37313/ using document root <none> and password file <none>
I20260620 20:16:34.326556 23978 fs_manager.cc:359] Metadata directory not provided
I20260620 20:16:34.326694 23978 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:16:34.330363 23978 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.003s
I20260620 20:16:34.332440 24189 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:16:34.332890 23978 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20260620 20:16:34.333062 23978 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986536687152-17252-0/minicluster-data/master-2/data,/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986536687152-17252-0/minicluster-data/master-2/wal
uuid: "5c48720af35c483da74e1243b72c87c3"
format_stamp: "Formatted at 2026-06-20 20:16:33 on dist-test-slave-2hpg"
server_key: "260b8aae21f9e0414f4f057cb096f035"
server_key_iv: "ef3f6485daa0d7d697c707ec073b966f"
server_key_version: "encryptionkey@0"
I20260620 20:16:34.333339 23978 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986536687152-17252-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986536687152-17252-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986536687152-17252-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:16:34.352169 23978 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:16:34.356328 23978 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:16:34.361851 17252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu with pid 23476
I20260620 20:16:34.386415 17252 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
/tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986536687152-17252-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986536687152-17252-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986536687152-17252-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986536687152-17252-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.217.1:34627
--local_ip_for_outbound_sockets=127.16.217.1
--tserver_master_addrs=127.16.217.62:36319
--webserver_port=45527
--webserver_interface=127.16.217.1
--keytab_file=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986536687152-17252-0/krb5kdc/oryx_127.16.217.1.keytab
--principal=oryx/127.16.217.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.217.20:38255
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.16.217.62:36319,127.16.217.61:33441
--tserver_master_addrs=127.16.217.62:36319,127.16.217.61:33441,127.16.217.60:37035 with env {KRB5CCNAME=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986536687152-17252-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986536687152-17252-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986536687152-17252-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 20:16:34.389329 23978 rpc_server.cc:307] RPC server started. Bound to: 127.16.217.60:37035
I20260620 20:16:34.389578 24240 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.217.60:37035 every 8 connection(s)
I20260620 20:16:34.390843 23978 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986536687152-17252-0/minicluster-data/master-2/data/info.pb
I20260620 20:16:34.399341 24241 sys_catalog.cc:263] Verifying existing consensus state
I20260620 20:16:34.401728 24241 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5c48720af35c483da74e1243b72c87c3: Bootstrap starting.
I20260620 20:16:34.415669 24241 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5c48720af35c483da74e1243b72c87c3: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 20:16:34.416126 24241 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5c48720af35c483da74e1243b72c87c3: Bootstrap complete.
I20260620 20:16:34.417049 24241 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5c48720af35c483da74e1243b72c87c3 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "358753b177ca4441ba3b33b1e5fd7d66" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 36319 } } peers { permanent_uuid: "0fe54b7d8d1b498f81774ec398b1f6aa" member_type: VOTER last_known_addr { host: "127.16.217.61" port: 33441 } attrs { promote: false } } peers { permanent_uuid: "5c48720af35c483da74e1243b72c87c3" member_type: NON_VOTER last_known_addr { host: "127.16.217.60" port: 37035 } attrs { promote: true } }
I20260620 20:16:34.417290 24241 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5c48720af35c483da74e1243b72c87c3 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 5c48720af35c483da74e1243b72c87c3, State: Initialized, Role: LEARNER
I20260620 20:16:34.417534 24241 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5c48720af35c483da74e1243b72c87c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "358753b177ca4441ba3b33b1e5fd7d66" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 36319 } } peers { permanent_uuid: "0fe54b7d8d1b498f81774ec398b1f6aa" member_type: VOTER last_known_addr { host: "127.16.217.61" port: 33441 } attrs { promote: false } } peers { permanent_uuid: "5c48720af35c483da74e1243b72c87c3" member_type: NON_VOTER last_known_addr { host: "127.16.217.60" port: 37035 } attrs { promote: true } }
I20260620 20:16:34.418365 24245 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c48720af35c483da74e1243b72c87c3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "358753b177ca4441ba3b33b1e5fd7d66" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 36319 } } peers { permanent_uuid: "0fe54b7d8d1b498f81774ec398b1f6aa" member_type: VOTER last_known_addr { host: "127.16.217.61" port: 33441 } attrs { promote: false } } peers { permanent_uuid: "5c48720af35c483da74e1243b72c87c3" member_type: NON_VOTER last_known_addr { host: "127.16.217.60" port: 37035 } attrs { promote: true } } }
I20260620 20:16:34.418673 24241 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5c48720af35c483da74e1243b72c87c3 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:16:34.418850 24245 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c48720af35c483da74e1243b72c87c3 [sys.catalog]: This master's current role is: LEARNER
I20260620 20:16:34.427673 24256 catalog_manager.cc:1259] Loaded cluster ID: a002c6e7a28743d2b82d56e2d2592ade
I20260620 20:16:34.427784 24256 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 5c48720af35c483da74e1243b72c87c3: loading cluster ID for follower catalog manager: success
I20260620 20:16:34.430549 24256 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 5c48720af35c483da74e1243b72c87c3: acquiring CA information for follower catalog manager: success
I20260620 20:16:34.432289 24256 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 5c48720af35c483da74e1243b72c87c3: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 20:16:34.668388 24153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:34.661391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.60:37035 (local address 127.0.0.1:48960)
0620 20:16:34.661593 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0620 20:16:34.661618 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0620 20:16:34.661804 (+ 186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:16:34.662762 (+ 958us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:16:34.662766 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:16:34.662799 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:16:34.662954 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:16:34.662961 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:16:34.664990 (+ 2029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:16:34.664994 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:16:34.666321 (+ 1327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:16:34.666334 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:16:34.667810 (+ 1476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:16:34.667814 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:16:34.667946 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:34.667971 (+ 25us) client_negotiation.cc:769] Sending connection context
0620 20:16:34.668048 (+ 77us) client_negotiation.cc:240] Negotiation successful
0620 20:16:34.668147 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":65}
I20260620 20:16:34.668548 24257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:34.661512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48960 (local address 127.16.217.60:37035)
0620 20:16:34.662371 (+ 859us) server_negotiation.cc:206] Beginning negotiation
0620 20:16:34.662377 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:16:34.662403 (+ 26us) server_negotiation.cc:407] Connection header received
0620 20:16:34.662487 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:16:34.662492 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:16:34.662592 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:16:34.662737 (+ 145us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:16:34.663087 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:34.664853 (+ 1766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:34.666466 (+ 1613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:34.667628 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:34.667764 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:34.667978 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0620 20:16:34.668088 (+ 110us) server_negotiation.cc:299] Negotiation successful
0620 20:16:34.668208 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":689,"thread_start_us":522,"threads_started":1}
I20260620 20:16:34.669464 24216 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 5c48720af35c483da74e1243b72c87c3 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260620 20:16:34.670203 24245 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c48720af35c483da74e1243b72c87c3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 358753b177ca4441ba3b33b1e5fd7d66. Latest consensus state: current_term: 1 leader_uuid: "358753b177ca4441ba3b33b1e5fd7d66" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "358753b177ca4441ba3b33b1e5fd7d66" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 36319 } } peers { permanent_uuid: "0fe54b7d8d1b498f81774ec398b1f6aa" member_type: VOTER last_known_addr { host: "127.16.217.61" port: 33441 } attrs { promote: false } } peers { permanent_uuid: "5c48720af35c483da74e1243b72c87c3" member_type: NON_VOTER last_known_addr { host: "127.16.217.60" port: 37035 } attrs { promote: true } } }
I20260620 20:16:34.670506 24245 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c48720af35c483da74e1243b72c87c3 [sys.catalog]: This master's current role is: LEARNER
W20260620 20:16:34.784200 24243 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:16:34.784533 24243 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:16:34.784703 24243 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:16:34.792097 24243 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:16:34.792239 24243 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:16:34.792402 24243 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.217.1
I20260620 20:16:34.802031 24243 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.217.20:38255
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986536687152-17252-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986536687152-17252-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986536687152-17252-0/krb5kdc/oryx_127.16.217.1.keytab
--principal=oryx/127.16.217.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.217.1:34627
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986536687152-17252-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.217.1
--webserver_port=45527
--tserver_master_addrs=127.16.217.62:36319,127.16.217.61:33441,127.16.217.60:37035
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.217.1
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986536687152-17252-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:16:34.803362 24243 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:16:34.804674 24243 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:16:34.818302 24266 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:16:34.818216 24263 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:16:34.818245 24262 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:16:34.819085 24243 server_base.cc:1034] running on GCE node
I20260620 20:16:34.819782 24243 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:16:34.820715 24243 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:16:34.821961 24243 hybrid_clock.cc:648] HybridClock initialized: now 1781986594821907 us; error 34 us; skew 500 ppm
I20260620 20:16:34.822309 24243 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:16:34 dist-test-slave-2hpg krb5kdc[23370](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986594, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.217.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:16:34.825369 24243 init.cc:377] Logged in from keytab as oryx/127.16.217.1@KRBTEST.COM (short username oryx)
I20260620 20:16:34.826946 24243 webserver.cc:460] Webserver started at http://127.16.217.1:45527/ using document root <none> and password file <none>
I20260620 20:16:34.827632 24243 fs_manager.cc:359] Metadata directory not provided
I20260620 20:16:34.827785 24243 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:16:34.833755 24243 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260620 20:16:34.836818 24273 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:16:34.838382 24243 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260620 20:16:34.838567 24243 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986536687152-17252-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986536687152-17252-0/minicluster-data/ts-0/wal
uuid: "5280c2f4f3ef4c77890e5b5b7c7f78ae"
format_stamp: "Formatted at 2026-06-20 20:16:31 on dist-test-slave-2hpg"
server_key: "6d099e29af665cf2a57c8e3adfc04bb6"
server_key_iv: "534e6c51d628166c2528284312a65c15"
server_key_version: "encryptionkey@0"
I20260620 20:16:34.839784 24243 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986536687152-17252-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986536687152-17252-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986536687152-17252-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:16:34.877831 24243 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:16:34.881851 24243 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:16:34.882166 24243 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:16:34.883114 24243 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:16:34.884527 24243 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:16:34.884646 24243 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:16:34.884783 24243 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:16:34.884876 24243 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:16:34.920764 24243 rpc_server.cc:307] RPC server started. Bound to: 127.16.217.1:34627
I20260620 20:16:34.920850 24385 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.217.1:34627 every 8 connection(s)
I20260620 20:16:34.922473 24243 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986536687152-17252-0/minicluster-data/ts-0/data/info.pb
I20260620 20:16:34.925344 17252 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu as pid 24243
I20260620 20:16:34.925673 17252 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu pid 23610
Jun 20 20:16:34 dist-test-slave-2hpg krb5kdc[23370](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986590, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.16.217.2@KRBTEST.COM
Jun 20 20:16:34 dist-test-slave-2hpg krb5kdc[23370](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986594, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.217.1@KRBTEST.COM for oryx/127.16.217.60@KRBTEST.COM
Jun 20 20:16:34 dist-test-slave-2hpg krb5kdc[23370](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986594, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.217.1@KRBTEST.COM for oryx/127.16.217.61@KRBTEST.COM
Jun 20 20:16:34 dist-test-slave-2hpg krb5kdc[23370](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986594, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.217.1@KRBTEST.COM for oryx/127.16.217.62@KRBTEST.COM
I20260620 20:16:34.955224 24257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:34.935163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.217.1:35043 (local address 127.16.217.60:37035)
0620 20:16:34.935352 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0620 20:16:34.935358 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:16:34.937207 (+ 1849us) server_negotiation.cc:407] Connection header received
0620 20:16:34.937454 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:16:34.937459 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:16:34.937531 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:16:34.937594 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:16:34.938928 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:34.939927 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:34.942097 (+ 2170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:34.943148 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:34.943253 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:34.947889 (+ 4636us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:16:34.947894 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:16:34.947897 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:16:34.947927 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:16:34.951608 (+ 3681us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:16:34.952546 (+ 938us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:16:34.952551 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:16:34.952554 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:16:34.952702 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:16:34.953194 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:16:34.953200 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:16:34.953203 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:16:34.953571 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:16:34.953701 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0620 20:16:34.954667 (+ 966us) server_negotiation.cc:299] Negotiation successful
0620 20:16:34.954884 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260620 20:16:34.956532 24391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:34.926440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:36319 (local address 127.16.217.1:47959)
0620 20:16:34.927386 (+ 946us) negotiation.cc:107] Waiting for socket to connect
0620 20:16:34.927440 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0620 20:16:34.929978 (+ 2538us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:16:34.936063 (+ 6085us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:16:34.936076 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:16:34.938194 (+ 2118us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:16:34.938478 (+ 284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:16:34.938488 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:16:34.939932 (+ 1444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:16:34.939938 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:16:34.941906 (+ 1968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:16:34.941924 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:16:34.943231 (+ 1307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:16:34.943240 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:16:34.944252 (+ 1012us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:34.944972 (+ 720us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:16:34.944999 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:16:34.948878 (+ 3879us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:16:34.952488 (+ 3610us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:16:34.952495 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:16:34.952501 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:16:34.952978 (+ 477us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:16:34.953399 (+ 421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:16:34.953403 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:16:34.953406 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:16:34.953543 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:16:34.954223 (+ 680us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:16:34.954228 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:16:34.954395 (+ 167us) client_negotiation.cc:769] Sending connection context
0620 20:16:34.954555 (+ 160us) client_negotiation.cc:240] Negotiation successful
0620 20:16:34.955078 (+ 523us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":595,"mutex_wait_us":621,"thread_start_us":1378,"threads_started":1}
I20260620 20:16:34.957058 24396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:34.934753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.60:37035 (local address 127.16.217.1:35043)
0620 20:16:34.937112 (+ 2359us) negotiation.cc:107] Waiting for socket to connect
0620 20:16:34.937135 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 20:16:34.937310 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:16:34.937797 (+ 487us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:16:34.937802 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:16:34.938484 (+ 682us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:16:34.938771 (+ 287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:16:34.938782 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:16:34.940102 (+ 1320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:16:34.940107 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:16:34.941942 (+ 1835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:16:34.941955 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:16:34.943581 (+ 1626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:16:34.943586 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:16:34.943856 (+ 270us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:34.944745 (+ 889us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:16:34.944778 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:16:34.947665 (+ 2887us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:16:34.951746 (+ 4081us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:16:34.951755 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:16:34.951773 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:16:34.952377 (+ 604us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:16:34.952824 (+ 447us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:16:34.952829 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:16:34.952832 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:16:34.953074 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:16:34.953788 (+ 714us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:16:34.953796 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:16:34.954226 (+ 430us) client_negotiation.cc:769] Sending connection context
0620 20:16:34.956055 (+ 1829us) client_negotiation.cc:240] Negotiation successful
0620 20:16:34.956325 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2192,"mutex_wait_us":154,"thread_start_us":2058,"threads_started":1}
I20260620 20:16:34.957819 24393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:34.927066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.217.1:47959 (local address 127.16.217.62:36319)
0620 20:16:34.933925 (+ 6859us) server_negotiation.cc:206] Beginning negotiation
0620 20:16:34.933934 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 20:16:34.933955 (+ 21us) server_negotiation.cc:407] Connection header received
0620 20:16:34.934019 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:16:34.934027 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:16:34.934177 (+ 150us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:16:34.934678 (+ 501us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:16:34.938651 (+ 3973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:34.939774 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:34.942080 (+ 2306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:34.942816 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:34.942936 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:34.949095 (+ 6159us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:16:34.949101 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:16:34.949105 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:16:34.949143 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:16:34.952335 (+ 3192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:16:34.953114 (+ 779us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:16:34.953122 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:16:34.953125 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:16:34.953269 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:16:34.953670 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:16:34.953676 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:16:34.953679 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:16:34.954037 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:16:34.954168 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0620 20:16:34.957235 (+ 3067us) server_negotiation.cc:299] Negotiation successful
0620 20:16:34.957455 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6705,"thread_start_us":6631,"threads_started":1}
I20260620 20:16:34.958101 24394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:34.927282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48670 (local address 127.16.217.2:41479)
0620 20:16:34.934318 (+ 7036us) server_negotiation.cc:206] Beginning negotiation
0620 20:16:34.934330 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 20:16:34.934368 (+ 38us) server_negotiation.cc:407] Connection header received
0620 20:16:34.934451 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:16:34.934462 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:16:34.934768 (+ 306us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:16:34.934967 (+ 199us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:16:34.937252 (+ 2285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:34.938726 (+ 1474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:34.940587 (+ 1861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:34.941431 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:34.948145 (+ 6714us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:34.949390 (+ 1245us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:16:34.949403 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:16:34.949420 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:16:34.949466 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:16:34.954175 (+ 4709us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:16:34.955053 (+ 878us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:16:34.955062 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:16:34.955065 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:16:34.955221 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:16:34.955646 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:16:34.955653 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:16:34.955658 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:16:34.956069 (+ 411us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:16:34.956343 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0620 20:16:34.957435 (+ 1092us) server_negotiation.cc:299] Negotiation successful
0620 20:16:34.957689 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6866,"thread_start_us":6725,"threads_started":1}
I20260620 20:16:34.958743 24386 heartbeater.cc:347] Connected to a master server at 127.16.217.60:37035
I20260620 20:16:34.959066 24386 heartbeater.cc:464] Registering TS with master...
I20260620 20:16:34.959426 23733 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:16:34.959659 24390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:34.926220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.217.1:56315 (local address 127.16.217.61:33441)
0620 20:16:34.935518 (+ 9298us) server_negotiation.cc:206] Beginning negotiation
0620 20:16:34.935522 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:16:34.935542 (+ 20us) server_negotiation.cc:407] Connection header received
0620 20:16:34.935586 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:16:34.935589 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:16:34.935649 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:16:34.935774 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:16:34.938255 (+ 2481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:34.939336 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:34.942101 (+ 2765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:34.943166 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:34.943270 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:34.948697 (+ 5427us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:16:34.948705 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:16:34.948713 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:16:34.949389 (+ 676us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:16:34.955460 (+ 6071us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:16:34.956239 (+ 779us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:16:34.956247 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:16:34.956250 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:16:34.956399 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:16:34.956816 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:16:34.956821 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:16:34.956824 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:16:34.957153 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:16:34.957298 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0620 20:16:34.959219 (+ 1921us) server_negotiation.cc:299] Negotiation successful
0620 20:16:34.959377 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9200,"thread_start_us":9109,"threads_started":1}
I20260620 20:16:34.960037 24386 heartbeater.cc:511] Master 127.16.217.60:37035 requested a full tablet report, sending...
I20260620 20:16:34.960111 24395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:34.927448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.61:33441 (local address 127.16.217.1:56315)
0620 20:16:34.928091 (+ 643us) negotiation.cc:107] Waiting for socket to connect
0620 20:16:34.928108 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 20:16:34.929978 (+ 1870us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:16:34.936063 (+ 6085us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:16:34.936076 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:16:34.937006 (+ 930us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:16:34.938022 (+ 1016us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:16:34.938052 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:16:34.939518 (+ 1466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:16:34.939524 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:16:34.941944 (+ 2420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:16:34.941955 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:16:34.943577 (+ 1622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:16:34.943583 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:16:34.944390 (+ 807us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:34.945480 (+ 1090us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:16:34.945501 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:16:34.948162 (+ 2661us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:16:34.955610 (+ 7448us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:16:34.955617 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:16:34.955620 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:16:34.956114 (+ 494us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:16:34.956528 (+ 414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:16:34.956532 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:16:34.956535 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:16:34.956693 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:16:34.957371 (+ 678us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:16:34.957375 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:16:34.957517 (+ 142us) client_negotiation.cc:769] Sending connection context
0620 20:16:34.957666 (+ 149us) client_negotiation.cc:240] Negotiation successful
0620 20:16:34.958865 (+ 1199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":484,"mutex_wait_us":697,"spinlock_wait_cycles":27904,"thread_start_us":313,"threads_started":1}
I20260620 20:16:34.960973 24388 heartbeater.cc:347] Connected to a master server at 127.16.217.61:33441
I20260620 20:16:34.961036 24387 heartbeater.cc:347] Connected to a master server at 127.16.217.62:36319
I20260620 20:16:34.961107 24388 heartbeater.cc:464] Registering TS with master...
I20260620 20:16:34.961186 24387 heartbeater.cc:464] Registering TS with master...
I20260620 20:16:34.961681 24388 heartbeater.cc:511] Master 127.16.217.61:33441 requested a full tablet report, sending...
I20260620 20:16:34.963222 24015 ts_manager.cc:194] Registered new tserver with Master: 5280c2f4f3ef4c77890e5b5b7c7f78ae (127.16.217.1:34627)
I20260620 20:16:34.963392 24206 ts_manager.cc:194] Registered new tserver with Master: 5280c2f4f3ef4c77890e5b5b7c7f78ae (127.16.217.1:34627)
I20260620 20:16:34.964471 24387 heartbeater.cc:511] Master 127.16.217.62:36319 requested a full tablet report, sending...
I20260620 20:16:34.965643 23415 ts_manager.cc:194] Re-registered known tserver with Master: 5280c2f4f3ef4c77890e5b5b7c7f78ae (127.16.217.1:34627)
I20260620 20:16:34.966643 23415 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.16.217.1@KRBTEST.COM'} at 127.16.217.1:47959
I20260620 20:16:35.044093 17252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu with pid 23610
I20260620 20:16:35.060504 17252 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
/tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986536687152-17252-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986536687152-17252-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986536687152-17252-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986536687152-17252-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.217.2:41479
--local_ip_for_outbound_sockets=127.16.217.2
--tserver_master_addrs=127.16.217.62:36319
--webserver_port=41695
--webserver_interface=127.16.217.2
--keytab_file=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986536687152-17252-0/krb5kdc/oryx_127.16.217.2.keytab
--principal=oryx/127.16.217.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.217.20:38255
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.16.217.62:36319,127.16.217.61:33441
--tserver_master_addrs=127.16.217.62:36319,127.16.217.61:33441,127.16.217.60:37035 with env {KRB5CCNAME=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986536687152-17252-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986536687152-17252-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986536687152-17252-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 20:16:35.129078 24171 sys_catalog.cc:455] T 00000000000000000000000000000000 P 358753b177ca4441ba3b33b1e5fd7d66 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "358753b177ca4441ba3b33b1e5fd7d66" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "358753b177ca4441ba3b33b1e5fd7d66" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 36319 } } peers { permanent_uuid: "0fe54b7d8d1b498f81774ec398b1f6aa" member_type: VOTER last_known_addr { host: "127.16.217.61" port: 33441 } attrs { promote: false } } peers { permanent_uuid: "5c48720af35c483da74e1243b72c87c3" member_type: NON_VOTER last_known_addr { host: "127.16.217.60" port: 37035 } attrs { promote: true } } }
I20260620 20:16:35.129527 24171 sys_catalog.cc:458] T 00000000000000000000000000000000 P 358753b177ca4441ba3b33b1e5fd7d66 [sys.catalog]: This master's current role is: LEADER
I20260620 20:16:35.130470 24171 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 358753b177ca4441ba3b33b1e5fd7d66: attempting to promote NON_VOTER 5c48720af35c483da74e1243b72c87c3 to VOTER
I20260620 20:16:35.131019 24171 consensus_queue.cc:237] T 00000000000000000000000000000000 P 358753b177ca4441ba3b33b1e5fd7d66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "358753b177ca4441ba3b33b1e5fd7d66" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 36319 } } peers { permanent_uuid: "0fe54b7d8d1b498f81774ec398b1f6aa" member_type: VOTER last_known_addr { host: "127.16.217.61" port: 33441 } attrs { promote: false } } peers { permanent_uuid: "5c48720af35c483da74e1243b72c87c3" member_type: VOTER last_known_addr { host: "127.16.217.60" port: 37035 } attrs { promote: false } }
I20260620 20:16:35.132664 24025 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 0fe54b7d8d1b498f81774ec398b1f6aa [term 1 FOLLOWER]: Refusing update from remote peer 358753b177ca4441ba3b33b1e5fd7d66: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260620 20:16:35.132870 24216 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 5c48720af35c483da74e1243b72c87c3 [term 1 LEARNER]: Refusing update from remote peer 358753b177ca4441ba3b33b1e5fd7d66: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260620 20:16:35.133283 24171 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 358753b177ca4441ba3b33b1e5fd7d66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0fe54b7d8d1b498f81774ec398b1f6aa" member_type: VOTER last_known_addr { host: "127.16.217.61" port: 33441 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260620 20:16:35.133643 24399 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 358753b177ca4441ba3b33b1e5fd7d66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5c48720af35c483da74e1243b72c87c3" member_type: VOTER last_known_addr { host: "127.16.217.60" port: 37035 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260620 20:16:35.136569 24025 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 0fe54b7d8d1b498f81774ec398b1f6aa [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 5c48720af35c483da74e1243b72c87c3 (127.16.217.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "358753b177ca4441ba3b33b1e5fd7d66" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 36319 } } peers { permanent_uuid: "0fe54b7d8d1b498f81774ec398b1f6aa" member_type: VOTER last_known_addr { host: "127.16.217.61" port: 33441 } attrs { promote: false } } peers { permanent_uuid: "5c48720af35c483da74e1243b72c87c3" member_type: VOTER last_known_addr { host: "127.16.217.60" port: 37035 } attrs { promote: false } } }
I20260620 20:16:35.136029 24399 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 358753b177ca4441ba3b33b1e5fd7d66 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 5c48720af35c483da74e1243b72c87c3 (127.16.217.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "358753b177ca4441ba3b33b1e5fd7d66" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 36319 } } peers { permanent_uuid: "0fe54b7d8d1b498f81774ec398b1f6aa" member_type: VOTER last_known_addr { host: "127.16.217.61" port: 33441 } attrs { promote: false } } peers { permanent_uuid: "5c48720af35c483da74e1243b72c87c3" member_type: VOTER last_known_addr { host: "127.16.217.60" port: 37035 } attrs { promote: false } } }
I20260620 20:16:35.142856 24403 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0fe54b7d8d1b498f81774ec398b1f6aa [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "358753b177ca4441ba3b33b1e5fd7d66" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "358753b177ca4441ba3b33b1e5fd7d66" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 36319 } } peers { permanent_uuid: "0fe54b7d8d1b498f81774ec398b1f6aa" member_type: VOTER last_known_addr { host: "127.16.217.61" port: 33441 } attrs { promote: false } } peers { permanent_uuid: "5c48720af35c483da74e1243b72c87c3" member_type: VOTER last_known_addr { host: "127.16.217.60" port: 37035 } attrs { promote: false } } }
I20260620 20:16:35.143335 24403 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0fe54b7d8d1b498f81774ec398b1f6aa [sys.catalog]: This master's current role is: FOLLOWER
I20260620 20:16:35.146031 24400 sys_catalog.cc:455] T 00000000000000000000000000000000 P 358753b177ca4441ba3b33b1e5fd7d66 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "358753b177ca4441ba3b33b1e5fd7d66" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "358753b177ca4441ba3b33b1e5fd7d66" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 36319 } } peers { permanent_uuid: "0fe54b7d8d1b498f81774ec398b1f6aa" member_type: VOTER last_known_addr { host: "127.16.217.61" port: 33441 } attrs { promote: false } } peers { permanent_uuid: "5c48720af35c483da74e1243b72c87c3" member_type: VOTER last_known_addr { host: "127.16.217.60" port: 37035 } attrs { promote: false } } }
I20260620 20:16:35.146332 24400 sys_catalog.cc:458] T 00000000000000000000000000000000 P 358753b177ca4441ba3b33b1e5fd7d66 [sys.catalog]: This master's current role is: LEADER
I20260620 20:16:35.146667 24400 sys_catalog.cc:455] T 00000000000000000000000000000000 P 358753b177ca4441ba3b33b1e5fd7d66 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "358753b177ca4441ba3b33b1e5fd7d66" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "358753b177ca4441ba3b33b1e5fd7d66" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 36319 } } peers { permanent_uuid: "0fe54b7d8d1b498f81774ec398b1f6aa" member_type: VOTER last_known_addr { host: "127.16.217.61" port: 33441 } attrs { promote: false } } peers { permanent_uuid: "5c48720af35c483da74e1243b72c87c3" member_type: VOTER last_known_addr { host: "127.16.217.60" port: 37035 } attrs { promote: false } } }
I20260620 20:16:35.146315 24216 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 5c48720af35c483da74e1243b72c87c3 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 5c48720af35c483da74e1243b72c87c3 (127.16.217.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "358753b177ca4441ba3b33b1e5fd7d66" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 36319 } } peers { permanent_uuid: "0fe54b7d8d1b498f81774ec398b1f6aa" member_type: VOTER last_known_addr { host: "127.16.217.61" port: 33441 } attrs { promote: false } } peers { permanent_uuid: "5c48720af35c483da74e1243b72c87c3" member_type: VOTER last_known_addr { host: "127.16.217.60" port: 37035 } attrs { promote: false } } }
I20260620 20:16:35.146867 24400 sys_catalog.cc:458] T 00000000000000000000000000000000 P 358753b177ca4441ba3b33b1e5fd7d66 [sys.catalog]: This master's current role is: LEADER
I20260620 20:16:35.147209 24400 sys_catalog.cc:455] T 00000000000000000000000000000000 P 358753b177ca4441ba3b33b1e5fd7d66 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "358753b177ca4441ba3b33b1e5fd7d66" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "358753b177ca4441ba3b33b1e5fd7d66" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 36319 } } peers { permanent_uuid: "0fe54b7d8d1b498f81774ec398b1f6aa" member_type: VOTER last_known_addr { host: "127.16.217.61" port: 33441 } attrs { promote: false } } peers { permanent_uuid: "5c48720af35c483da74e1243b72c87c3" member_type: VOTER last_known_addr { host: "127.16.217.60" port: 37035 } attrs { promote: false } } }
I20260620 20:16:35.147380 24400 sys_catalog.cc:458] T 00000000000000000000000000000000 P 358753b177ca4441ba3b33b1e5fd7d66 [sys.catalog]: This master's current role is: LEADER
I20260620 20:16:35.150612 24245 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c48720af35c483da74e1243b72c87c3 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "358753b177ca4441ba3b33b1e5fd7d66" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "358753b177ca4441ba3b33b1e5fd7d66" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 36319 } } peers { permanent_uuid: "0fe54b7d8d1b498f81774ec398b1f6aa" member_type: VOTER last_known_addr { host: "127.16.217.61" port: 33441 } attrs { promote: false } } peers { permanent_uuid: "5c48720af35c483da74e1243b72c87c3" member_type: VOTER last_known_addr { host: "127.16.217.60" port: 37035 } attrs { promote: false } } }
I20260620 20:16:35.150854 24245 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c48720af35c483da74e1243b72c87c3 [sys.catalog]: This master's current role is: FOLLOWER
W20260620 20:16:35.402437 24398 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:16:35.402697 24398 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:16:35.402799 24398 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:16:35.410024 24398 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:16:35.410130 24398 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:16:35.410240 24398 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.217.2
I20260620 20:16:35.420042 24398 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.217.20:38255
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986536687152-17252-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986536687152-17252-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986536687152-17252-0/krb5kdc/oryx_127.16.217.2.keytab
--principal=oryx/127.16.217.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.217.2:41479
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986536687152-17252-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.217.2
--webserver_port=41695
--tserver_master_addrs=127.16.217.62:36319,127.16.217.61:33441,127.16.217.60:37035
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.217.2
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986536687152-17252-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:16:35.421294 24398 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:16:35.422667 24398 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:16:35.435727 24408 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:16:35.435910 24409 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:16:35.435932 24411 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:16:35.437198 24398 server_base.cc:1034] running on GCE node
I20260620 20:16:35.437776 24398 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:16:35.438738 24398 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:16:35.439941 24398 hybrid_clock.cc:648] HybridClock initialized: now 1781986595439890 us; error 48 us; skew 500 ppm
I20260620 20:16:35.440255 24398 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:16:35 dist-test-slave-2hpg krb5kdc[23370](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986595, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.217.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:16:35.443373 24398 init.cc:377] Logged in from keytab as oryx/127.16.217.2@KRBTEST.COM (short username oryx)
I20260620 20:16:35.444897 24398 webserver.cc:460] Webserver started at http://127.16.217.2:41695/ using document root <none> and password file <none>
I20260620 20:16:35.445554 24398 fs_manager.cc:359] Metadata directory not provided
I20260620 20:16:35.445741 24398 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:16:35.451058 24398 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260620 20:16:35.453675 24418 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:16:35.455019 24398 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 20:16:35.455186 24398 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986536687152-17252-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986536687152-17252-0/minicluster-data/ts-1/wal
uuid: "33fb52c1b00347a8ac5db344e2959cac"
format_stamp: "Formatted at 2026-06-20 20:16:32 on dist-test-slave-2hpg"
server_key: "df26fbfc2e9cd40fc4b1ae167fdaaff2"
server_key_iv: "0d9acd9345547d7dd4b0503c37a03bce"
server_key_version: "encryptionkey@0"
I20260620 20:16:35.456132 24398 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986536687152-17252-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986536687152-17252-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986536687152-17252-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:16:35.469735 24398 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:16:35.473773 24398 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:16:35.474095 24398 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:16:35.475095 24398 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:16:35.476526 24398 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:16:35.476651 24398 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:16:35.476766 24398 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:16:35.476854 24398 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:16:35.509835 24398 rpc_server.cc:307] RPC server started. Bound to: 127.16.217.2:41479
I20260620 20:16:35.509927 24532 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.217.2:41479 every 8 connection(s)
I20260620 20:16:35.511313 24398 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986536687152-17252-0/minicluster-data/ts-1/data/info.pb
I20260620 20:16:35.520879 17252 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu as pid 24398
I20260620 20:16:35.521199 17252 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu pid 23743
Jun 20 20:16:35 dist-test-slave-2hpg krb5kdc[23370](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986595, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.217.2@KRBTEST.COM for oryx/127.16.217.62@KRBTEST.COM
Jun 20 20:16:35 dist-test-slave-2hpg krb5kdc[23370](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986595, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.217.2@KRBTEST.COM for oryx/127.16.217.61@KRBTEST.COM
Jun 20 20:16:35 dist-test-slave-2hpg krb5kdc[23370](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986595, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.217.2@KRBTEST.COM for oryx/127.16.217.60@KRBTEST.COM
I20260620 20:16:35.542609 24538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:35.514280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.217.2:55463 (local address 127.16.217.62:36319)
0620 20:16:35.515355 (+ 1075us) server_negotiation.cc:206] Beginning negotiation
0620 20:16:35.515362 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:16:35.517965 (+ 2603us) server_negotiation.cc:407] Connection header received
0620 20:16:35.520191 (+ 2226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:16:35.520197 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:16:35.520304 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:16:35.520383 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:16:35.523293 (+ 2910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:35.524377 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:35.527094 (+ 2717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:35.527825 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:35.527941 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:35.534298 (+ 6357us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:16:35.534309 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:16:35.534315 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:16:35.534374 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:16:35.538728 (+ 4354us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:16:35.539733 (+ 1005us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:16:35.539738 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:16:35.539742 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:16:35.539884 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:16:35.540401 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:16:35.540409 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:16:35.540415 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:16:35.540793 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:16:35.540961 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0620 20:16:35.541976 (+ 1015us) server_negotiation.cc:299] Negotiation successful
0620 20:16:35.542218 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":940,"thread_start_us":1215,"threads_started":1}
I20260620 20:16:35.542798 24539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:35.514625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.217.2:37639 (local address 127.16.217.61:33441)
0620 20:16:35.516011 (+ 1386us) server_negotiation.cc:206] Beginning negotiation
0620 20:16:35.516020 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 20:16:35.520319 (+ 4299us) server_negotiation.cc:407] Connection header received
0620 20:16:35.520586 (+ 267us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:16:35.520591 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:16:35.520737 (+ 146us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:16:35.520815 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:16:35.524547 (+ 3732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:35.525629 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:35.527966 (+ 2337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:35.528675 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:35.528775 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:35.534296 (+ 5521us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:16:35.534309 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:16:35.534315 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:16:35.534378 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:16:35.539381 (+ 5003us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:16:35.540222 (+ 841us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:16:35.540227 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:16:35.540231 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:16:35.540375 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:16:35.540809 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:16:35.540813 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:16:35.540819 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:16:35.541192 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:16:35.541327 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0620 20:16:35.541979 (+ 652us) server_negotiation.cc:299] Negotiation successful
0620 20:16:35.542474 (+ 495us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1227,"thread_start_us":1138,"threads_started":1}
I20260620 20:16:35.545428 24540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:35.515287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.61:33441 (local address 127.16.217.2:37639)
0620 20:16:35.520232 (+ 4945us) negotiation.cc:107] Waiting for socket to connect
0620 20:16:35.520249 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 20:16:35.520441 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:16:35.521011 (+ 570us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:16:35.521015 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:16:35.522261 (+ 1246us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:16:35.524380 (+ 2119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:16:35.524403 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:16:35.525806 (+ 1403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:16:35.525811 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:16:35.527765 (+ 1954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:16:35.527780 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:16:35.529163 (+ 1383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:16:35.529170 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:16:35.529306 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:35.529931 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:16:35.529959 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:16:35.533659 (+ 3700us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:16:35.539536 (+ 5877us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:16:35.539540 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:16:35.539544 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:16:35.540079 (+ 535us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:16:35.540519 (+ 440us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:16:35.540524 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:16:35.540527 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:16:35.540675 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:16:35.541398 (+ 723us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:16:35.541402 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:16:35.541561 (+ 159us) client_negotiation.cc:769] Sending connection context
0620 20:16:35.543008 (+ 1447us) client_negotiation.cc:240] Negotiation successful
0620 20:16:35.543651 (+ 643us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4830,"mutex_wait_us":1679,"spinlock_wait_cycles":5248,"thread_start_us":1339,"threads_started":1}
I20260620 20:16:35.545446 24542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:35.514849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:36319 (local address 127.16.217.2:55463)
0620 20:16:35.517022 (+ 2173us) negotiation.cc:107] Waiting for socket to connect
0620 20:16:35.517074 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0620 20:16:35.519793 (+ 2719us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:16:35.520933 (+ 1140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:16:35.520948 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:16:35.522261 (+ 1313us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:16:35.523107 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:16:35.523133 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:16:35.524576 (+ 1443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:16:35.524581 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:16:35.526914 (+ 2333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:16:35.526929 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:16:35.528324 (+ 1395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:16:35.528332 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:16:35.528659 (+ 327us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:35.529618 (+ 959us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:16:35.529656 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:16:35.533991 (+ 4335us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:16:35.538901 (+ 4910us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:16:35.538913 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:16:35.538936 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:16:35.539575 (+ 639us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:16:35.540046 (+ 471us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:16:35.540050 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:16:35.540053 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:16:35.540279 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:16:35.541049 (+ 770us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:16:35.541057 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:16:35.541528 (+ 471us) client_negotiation.cc:769] Sending connection context
0620 20:16:35.542896 (+ 1368us) client_negotiation.cc:240] Negotiation successful
0620 20:16:35.543527 (+ 631us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1888,"mutex_wait_us":51,"spinlock_wait_cycles":768,"thread_start_us":4991,"threads_started":1}
I20260620 20:16:35.547096 24535 heartbeater.cc:347] Connected to a master server at 127.16.217.61:33441
I20260620 20:16:35.547463 24535 heartbeater.cc:464] Registering TS with master...
I20260620 20:16:35.548368 24534 heartbeater.cc:347] Connected to a master server at 127.16.217.62:36319
I20260620 20:16:35.548424 24535 heartbeater.cc:511] Master 127.16.217.61:33441 requested a full tablet report, sending...
I20260620 20:16:35.548519 24534 heartbeater.cc:464] Registering TS with master...
Jun 20 20:16:35 dist-test-slave-2hpg krb5kdc[23370](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986590, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.16.217.3@KRBTEST.COM
I20260620 20:16:35.549015 24534 heartbeater.cc:511] Master 127.16.217.62:36319 requested a full tablet report, sending...
I20260620 20:16:35.550280 23415 ts_manager.cc:194] Re-registered known tserver with Master: 33fb52c1b00347a8ac5db344e2959cac (127.16.217.2:41479)
I20260620 20:16:35.550318 24015 ts_manager.cc:194] Registered new tserver with Master: 33fb52c1b00347a8ac5db344e2959cac (127.16.217.2:41479)
I20260620 20:16:35.550757 24541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:35.514849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.60:37035 (local address 127.16.217.2:51305)
0620 20:16:35.519570 (+ 4721us) negotiation.cc:107] Waiting for socket to connect
0620 20:16:35.519593 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 20:16:35.520220 (+ 627us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:16:35.527908 (+ 7688us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:16:35.527917 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:16:35.529627 (+ 1710us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:16:35.530084 (+ 457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:16:35.530102 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:16:35.532541 (+ 2439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:16:35.532546 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:16:35.536151 (+ 3605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:16:35.536167 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:16:35.537537 (+ 1370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:16:35.537546 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:16:35.537744 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:35.538327 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:16:35.538349 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:16:35.541037 (+ 2688us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:16:35.545498 (+ 4461us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:16:35.545505 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:16:35.545510 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:16:35.547571 (+ 2061us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:16:35.548042 (+ 471us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:16:35.548047 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:16:35.548054 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:16:35.548213 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:16:35.548920 (+ 707us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:16:35.548925 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:16:35.549066 (+ 141us) client_negotiation.cc:769] Sending connection context
0620 20:16:35.549966 (+ 900us) client_negotiation.cc:240] Negotiation successful
0620 20:16:35.550305 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4561,"mutex_wait_us":26,"spinlock_wait_cycles":73984,"thread_start_us":4279,"threads_started":1}
I20260620 20:16:35.551309 24537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:35.514136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.217.2:51305 (local address 127.16.217.60:37035)
0620 20:16:35.518360 (+ 4224us) server_negotiation.cc:206] Beginning negotiation
0620 20:16:35.518368 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 20:16:35.519741 (+ 1373us) server_negotiation.cc:407] Connection header received
0620 20:16:35.521114 (+ 1373us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:16:35.521126 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:16:35.521327 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:16:35.521419 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:16:35.530998 (+ 9579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:35.532289 (+ 1291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:35.536315 (+ 4026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:35.537055 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:35.537167 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:35.541259 (+ 4092us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:16:35.541265 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:16:35.541272 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:16:35.541314 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:16:35.545290 (+ 3976us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:16:35.547707 (+ 2417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:16:35.547745 (+ 38us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:16:35.547748 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:16:35.547899 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:16:35.548352 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:16:35.548357 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:16:35.548360 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:16:35.548692 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:16:35.548836 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0620 20:16:35.549386 (+ 550us) server_negotiation.cc:299] Negotiation successful
0620 20:16:35.549596 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4080,"thread_start_us":3957,"threads_started":1}
I20260620 20:16:35.551420 23415 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.16.217.2@KRBTEST.COM'} at 127.16.217.2:55463
I20260620 20:16:35.552135 24533 heartbeater.cc:347] Connected to a master server at 127.16.217.60:37035
I20260620 20:16:35.552271 24533 heartbeater.cc:464] Registering TS with master...
I20260620 20:16:35.552721 24533 heartbeater.cc:511] Master 127.16.217.60:37035 requested a full tablet report, sending...
I20260620 20:16:35.553979 24206 ts_manager.cc:194] Registered new tserver with Master: 33fb52c1b00347a8ac5db344e2959cac (127.16.217.2:41479)
I20260620 20:16:35.557780 24544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:35.522313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53898 (local address 127.16.217.3:46165)
0620 20:16:35.535523 (+ 13210us) server_negotiation.cc:206] Beginning negotiation
0620 20:16:35.535532 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 20:16:35.535563 (+ 31us) server_negotiation.cc:407] Connection header received
0620 20:16:35.535633 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:16:35.535642 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:16:35.535896 (+ 254us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:16:35.536066 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:16:35.538941 (+ 2875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:35.540607 (+ 1666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:35.542701 (+ 2094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:35.543752 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:35.543853 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:35.550589 (+ 6736us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:16:35.550604 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:16:35.550620 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:16:35.550662 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:16:35.554656 (+ 3994us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:16:35.555338 (+ 682us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:16:35.555345 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:16:35.555347 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:16:35.555478 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:16:35.555858 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:16:35.555866 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:16:35.555868 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:16:35.556170 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:16:35.556465 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0620 20:16:35.556790 (+ 325us) server_negotiation.cc:299] Negotiation successful
0620 20:16:35.557054 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13015,"thread_start_us":13047,"threads_started":1}
I20260620 20:16:35.559000 23866 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:16:35.641535 17252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu with pid 23743
I20260620 20:16:35.658859 17252 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
/tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986536687152-17252-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986536687152-17252-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986536687152-17252-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986536687152-17252-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.217.3:46165
--local_ip_for_outbound_sockets=127.16.217.3
--tserver_master_addrs=127.16.217.62:36319
--webserver_port=46693
--webserver_interface=127.16.217.3
--keytab_file=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986536687152-17252-0/krb5kdc/oryx_127.16.217.3.keytab
--principal=oryx/127.16.217.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.217.20:38255
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.16.217.62:36319,127.16.217.61:33441
--tserver_master_addrs=127.16.217.62:36319,127.16.217.61:33441,127.16.217.60:37035 with env {KRB5CCNAME=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986536687152-17252-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986536687152-17252-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986536687152-17252-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 20:16:35.970293 24387 heartbeater.cc:503] Master 127.16.217.62:36319 was elected leader, sending a full tablet report...
W20260620 20:16:35.986897 24546 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:16:35.987222 24546 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:16:35.987403 24546 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:16:35.994788 24546 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:16:35.994935 24546 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:16:35.995115 24546 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.217.3
I20260620 20:16:36.004793 24546 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.217.20:38255
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986536687152-17252-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986536687152-17252-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986536687152-17252-0/krb5kdc/oryx_127.16.217.3.keytab
--principal=oryx/127.16.217.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.217.3:46165
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986536687152-17252-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.217.3
--webserver_port=46693
--tserver_master_addrs=127.16.217.62:36319,127.16.217.61:33441,127.16.217.60:37035
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.217.3
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986536687152-17252-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:16:36.006192 24546 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:16:36.007550 24546 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:16:36.021163 24551 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:16:36.021194 24552 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:16:36.021130 24554 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:16:36.022823 24546 server_base.cc:1034] running on GCE node
I20260620 20:16:36.023587 24546 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:16:36.024598 24546 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:16:36.025847 24546 hybrid_clock.cc:648] HybridClock initialized: now 1781986596025786 us; error 65 us; skew 500 ppm
I20260620 20:16:36.026257 24546 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:16:36 dist-test-slave-2hpg krb5kdc[23370](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986596, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.217.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:16:36.029526 24546 init.cc:377] Logged in from keytab as oryx/127.16.217.3@KRBTEST.COM (short username oryx)
I20260620 20:16:36.031157 24546 webserver.cc:460] Webserver started at http://127.16.217.3:46693/ using document root <none> and password file <none>
I20260620 20:16:36.031898 24546 fs_manager.cc:359] Metadata directory not provided
I20260620 20:16:36.032006 24546 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:16:36.037158 24546 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 20:16:36.039852 24561 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:16:36.041226 24546 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 20:16:36.041406 24546 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986536687152-17252-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986536687152-17252-0/minicluster-data/ts-2/wal
uuid: "b6d268e1e2e944af9e30e4b3e4b3e359"
format_stamp: "Formatted at 2026-06-20 20:16:32 on dist-test-slave-2hpg"
server_key: "a1edf7c2c082c607f2b1516616392cdf"
server_key_iv: "a8d4bd1d1e846917dbb01ca9fb55b819"
server_key_version: "encryptionkey@0"
I20260620 20:16:36.042441 24546 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986536687152-17252-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986536687152-17252-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986536687152-17252-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:16:36.073699 24546 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:16:36.078123 24546 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:16:36.078431 24546 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:16:36.079396 24546 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:16:36.080703 24546 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:16:36.080789 24546 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:16:36.080914 24546 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:16:36.081013 24546 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:16:36.111917 24546 rpc_server.cc:307] RPC server started. Bound to: 127.16.217.3:46165
I20260620 20:16:36.111968 24674 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.217.3:46165 every 8 connection(s)
I20260620 20:16:36.113348 24546 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986536687152-17252-0/minicluster-data/ts-2/data/info.pb
I20260620 20:16:36.119683 17252 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu as pid 24546
Jun 20 20:16:36 dist-test-slave-2hpg krb5kdc[23370](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986596, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.217.3@KRBTEST.COM for oryx/127.16.217.62@KRBTEST.COM
Jun 20 20:16:36 dist-test-slave-2hpg krb5kdc[23370](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986596, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.217.3@KRBTEST.COM for oryx/127.16.217.60@KRBTEST.COM
Jun 20 20:16:36 dist-test-slave-2hpg krb5kdc[23370](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986596, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.217.3@KRBTEST.COM for oryx/127.16.217.61@KRBTEST.COM
I20260620 20:16:36.162197 24682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:36.117822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.217.3:35663 (local address 127.16.217.61:33441)
0620 20:16:36.128187 (+ 10365us) server_negotiation.cc:206] Beginning negotiation
0620 20:16:36.128195 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 20:16:36.138141 (+ 9946us) server_negotiation.cc:407] Connection header received
0620 20:16:36.138632 (+ 491us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:16:36.138638 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:16:36.138731 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:16:36.138808 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:16:36.140420 (+ 1612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:36.141895 (+ 1475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:36.145062 (+ 3167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:36.145933 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:36.146085 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:36.152570 (+ 6485us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:16:36.152579 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:16:36.152586 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:16:36.152656 (+ 70us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:16:36.157294 (+ 4638us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:16:36.158418 (+ 1124us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:16:36.158427 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:16:36.158433 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:16:36.158621 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:16:36.159344 (+ 723us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:16:36.159352 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:16:36.159358 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:16:36.159767 (+ 409us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:16:36.160004 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0620 20:16:36.161224 (+ 1220us) server_negotiation.cc:299] Negotiation successful
0620 20:16:36.161620 (+ 396us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8802,"thread_start_us":8713,"threads_started":1}
I20260620 20:16:36.162359 24679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:36.116125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.217.3:56665 (local address 127.16.217.62:36319)
0620 20:16:36.141619 (+ 25494us) server_negotiation.cc:206] Beginning negotiation
0620 20:16:36.141625 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:16:36.141644 (+ 19us) server_negotiation.cc:407] Connection header received
0620 20:16:36.141700 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:16:36.141705 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:16:36.141785 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:16:36.143128 (+ 1343us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:16:36.143288 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:36.144421 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:36.146263 (+ 1842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:36.146855 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:36.146974 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:36.153074 (+ 6100us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:16:36.153081 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:16:36.153087 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:16:36.153129 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:16:36.158116 (+ 4987us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:16:36.159433 (+ 1317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:16:36.159438 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:16:36.159441 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:16:36.159572 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:16:36.160897 (+ 1325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:16:36.160902 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:16:36.160905 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:16:36.161238 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:16:36.161382 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0620 20:16:36.161786 (+ 404us) server_negotiation.cc:299] Negotiation successful
0620 20:16:36.162023 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25364,"thread_start_us":26842,"threads_started":1}
I20260620 20:16:36.163704 24685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:36.117035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:36319 (local address 127.16.217.3:56665)
0620 20:16:36.127011 (+ 9976us) negotiation.cc:107] Waiting for socket to connect
0620 20:16:36.127073 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0620 20:16:36.131723 (+ 4650us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:16:36.141952 (+ 10229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:16:36.141956 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:16:36.142488 (+ 532us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:16:36.142880 (+ 392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:16:36.142897 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:16:36.144611 (+ 1714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:16:36.144615 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:16:36.146098 (+ 1483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:16:36.146113 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:16:36.147230 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:16:36.147235 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:16:36.147424 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:36.147925 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:16:36.147941 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:16:36.151448 (+ 3507us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:16:36.158322 (+ 6874us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:16:36.158328 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:16:36.158333 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:16:36.158788 (+ 455us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:16:36.159782 (+ 994us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:16:36.159786 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:16:36.159789 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:16:36.159949 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:16:36.161345 (+ 1396us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:16:36.161350 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:16:36.161527 (+ 177us) client_negotiation.cc:769] Sending connection context
0620 20:16:36.161659 (+ 132us) client_negotiation.cc:240] Negotiation successful
0620 20:16:36.162030 (+ 371us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8174,"thread_start_us":16953,"threads_started":1}
I20260620 20:16:36.164395 24683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:36.117035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.61:33441 (local address 127.16.217.3:35663)
0620 20:16:36.138005 (+ 20970us) negotiation.cc:107] Waiting for socket to connect
0620 20:16:36.138031 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0620 20:16:36.138469 (+ 438us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:16:36.139007 (+ 538us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:16:36.139011 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:16:36.139583 (+ 572us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:16:36.140168 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:16:36.140203 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:16:36.142131 (+ 1928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:16:36.142136 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:16:36.144824 (+ 2688us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:16:36.144849 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:16:36.146458 (+ 1609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:16:36.146466 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:16:36.146804 (+ 338us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:36.148174 (+ 1370us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:16:36.148203 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:16:36.152283 (+ 4080us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:16:36.157507 (+ 5224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:16:36.157515 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:16:36.157522 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:16:36.158234 (+ 712us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:16:36.158774 (+ 540us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:16:36.158782 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:16:36.158788 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:16:36.159163 (+ 375us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:16:36.160090 (+ 927us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:16:36.160098 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:16:36.160593 (+ 495us) client_negotiation.cc:769] Sending connection context
0620 20:16:36.160927 (+ 334us) client_negotiation.cc:240] Negotiation successful
0620 20:16:36.163313 (+ 2386us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20754,"mutex_wait_us":5175,"spinlock_wait_cycles":51200,"thread_start_us":3000,"threads_started":1}
I20260620 20:16:36.165302 24676 heartbeater.cc:347] Connected to a master server at 127.16.217.62:36319
I20260620 20:16:36.165628 24676 heartbeater.cc:464] Registering TS with master...
I20260620 20:16:36.165903 24681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:36.117491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.217.3:52443 (local address 127.16.217.60:37035)
0620 20:16:36.126398 (+ 8907us) server_negotiation.cc:206] Beginning negotiation
0620 20:16:36.126408 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 20:16:36.134540 (+ 8132us) server_negotiation.cc:407] Connection header received
0620 20:16:36.135559 (+ 1019us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:16:36.135569 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:16:36.135677 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:16:36.135786 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:16:36.140667 (+ 4881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:36.141970 (+ 1303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:36.145020 (+ 3050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:36.145674 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:36.145792 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:36.152784 (+ 6992us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:16:36.152794 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:16:36.152801 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:16:36.152855 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:16:36.156905 (+ 4050us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:16:36.158020 (+ 1115us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:16:36.158027 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:16:36.158033 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:16:36.158170 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:16:36.159200 (+ 1030us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:16:36.159207 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:16:36.159211 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:16:36.159589 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:16:36.159724 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0620 20:16:36.164495 (+ 4771us) server_negotiation.cc:299] Negotiation successful
0620 20:16:36.165519 (+ 1024us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8751,"thread_start_us":8705,"threads_started":1}
I20260620 20:16:36.166168 24677 heartbeater.cc:347] Connected to a master server at 127.16.217.61:33441
I20260620 20:16:36.166329 24677 heartbeater.cc:464] Registering TS with master...
I20260620 20:16:36.166664 24676 heartbeater.cc:511] Master 127.16.217.62:36319 requested a full tablet report, sending...
I20260620 20:16:36.166832 24677 heartbeater.cc:511] Master 127.16.217.61:33441 requested a full tablet report, sending...
I20260620 20:16:36.166947 24680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:36.117749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.60:37035 (local address 127.16.217.3:52443)
0620 20:16:36.134412 (+ 16663us) negotiation.cc:107] Waiting for socket to connect
0620 20:16:36.134436 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 20:16:36.135198 (+ 762us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:16:36.136946 (+ 1748us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:16:36.136977 (+ 31us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:16:36.138955 (+ 1978us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:16:36.140474 (+ 1519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:16:36.140488 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:16:36.142135 (+ 1647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:16:36.142141 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:16:36.144811 (+ 2670us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:16:36.144827 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:16:36.146431 (+ 1604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:16:36.146487 (+ 56us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:16:36.146791 (+ 304us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:36.147673 (+ 882us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:16:36.147707 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:16:36.152521 (+ 4814us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:16:36.157084 (+ 4563us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:16:36.157095 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:16:36.157115 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:16:36.157806 (+ 691us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:16:36.158899 (+ 1093us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:16:36.158903 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:16:36.158906 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:16:36.159065 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:16:36.160083 (+ 1018us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:16:36.160092 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:16:36.160593 (+ 501us) client_negotiation.cc:769] Sending connection context
0620 20:16:36.160922 (+ 329us) client_negotiation.cc:240] Negotiation successful
0620 20:16:36.166383 (+ 5461us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16510,"mutex_wait_us":494,"spinlock_wait_cycles":4480,"thread_start_us":20397,"threads_started":1}
I20260620 20:16:36.168716 23415 ts_manager.cc:194] Re-registered known tserver with Master: b6d268e1e2e944af9e30e4b3e4b3e359 (127.16.217.3:46165)
I20260620 20:16:36.170148 23415 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.16.217.3@KRBTEST.COM'} at 127.16.217.3:56665
I20260620 20:16:36.170532 24675 heartbeater.cc:347] Connected to a master server at 127.16.217.60:37035
I20260620 20:16:36.170665 24675 heartbeater.cc:464] Registering TS with master...
I20260620 20:16:36.171037 24675 heartbeater.cc:511] Master 127.16.217.60:37035 requested a full tablet report, sending...
I20260620 20:16:36.171847 24015 ts_manager.cc:194] Registered new tserver with Master: b6d268e1e2e944af9e30e4b3e4b3e359 (127.16.217.3:46165)
I20260620 20:16:36.172191 24206 ts_manager.cc:194] Registered new tserver with Master: b6d268e1e2e944af9e30e4b3e4b3e359 (127.16.217.3:46165)
I20260620 20:16:36.184899 24689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:36.143945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59162 (local address 127.16.217.61:33441)
0620 20:16:36.155425 (+ 11480us) server_negotiation.cc:206] Beginning negotiation
0620 20:16:36.155431 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:16:36.158900 (+ 3469us) server_negotiation.cc:407] Connection header received
0620 20:16:36.158954 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:16:36.158960 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:16:36.159034 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:16:36.159143 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:16:36.169903 (+ 10760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:36.171155 (+ 1252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:36.172979 (+ 1824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:36.173712 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:36.173841 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:36.178455 (+ 4614us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:16:36.178462 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:16:36.178469 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:16:36.178513 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:16:36.181812 (+ 3299us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:16:36.182715 (+ 903us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:16:36.182721 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:16:36.182724 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:16:36.182880 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:16:36.183327 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:16:36.183332 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:16:36.183337 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:16:36.183701 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:16:36.183974 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0620 20:16:36.184341 (+ 367us) server_negotiation.cc:299] Negotiation successful
0620 20:16:36.184560 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11365,"thread_start_us":13088,"threads_started":1}
I20260620 20:16:36.188504 24691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:36.150287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50326 (local address 127.16.217.62:36319)
0620 20:16:36.154741 (+ 4454us) server_negotiation.cc:206] Beginning negotiation
0620 20:16:36.154747 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:16:36.170543 (+ 15796us) server_negotiation.cc:407] Connection header received
0620 20:16:36.170696 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:16:36.170739 (+ 43us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:16:36.170823 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:16:36.170930 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:16:36.175168 (+ 4238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:36.175973 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:36.177984 (+ 2011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:36.178397 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:36.178525 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:36.181238 (+ 2713us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:16:36.181242 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:16:36.181245 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:16:36.181275 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:16:36.184409 (+ 3134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:16:36.185811 (+ 1402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:16:36.185817 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:16:36.185821 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:16:36.186001 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:16:36.187160 (+ 1159us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:16:36.187165 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:16:36.187169 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:16:36.187430 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:16:36.187680 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0620 20:16:36.187978 (+ 298us) server_negotiation.cc:299] Negotiation successful
0620 20:16:36.188183 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4330,"thread_start_us":3576,"threads_started":1}
I20260620 20:16:36.191049 24681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:36.155490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48962 (local address 127.16.217.60:37035)
0620 20:16:36.166129 (+ 10639us) server_negotiation.cc:206] Beginning negotiation
0620 20:16:36.166135 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:16:36.171234 (+ 5099us) server_negotiation.cc:407] Connection header received
0620 20:16:36.171288 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:16:36.171318 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:16:36.171403 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:16:36.171494 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:16:36.177255 (+ 5761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:36.178380 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:36.180354 (+ 1974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:36.181021 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:36.181155 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:36.184792 (+ 3637us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:16:36.184796 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:16:36.184799 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:16:36.184828 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:16:36.188013 (+ 3185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:16:36.188769 (+ 756us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:16:36.188774 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:16:36.188777 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:16:36.188944 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:16:36.189354 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:16:36.189358 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:16:36.189360 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:16:36.189695 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:16:36.189975 (+ 280us) server_negotiation.cc:1035] Waiting for connection context
0620 20:16:36.190346 (+ 371us) server_negotiation.cc:299] Negotiation successful
0620 20:16:36.190591 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10533,"thread_start_us":13181,"threads_started":1}
I20260620 20:16:36.192832 23415 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50326:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 20:16:36.199610 23415 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 20:16:36.251210 24708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:36.222165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.3:46165 (local address 127.0.0.1:53902)
0620 20:16:36.234291 (+ 12126us) negotiation.cc:107] Waiting for socket to connect
0620 20:16:36.234314 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 20:16:36.234466 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:16:36.241715 (+ 7249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:16:36.241724 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:16:36.241763 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:16:36.242359 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:16:36.242376 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:16:36.245526 (+ 3150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:16:36.245535 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:16:36.248180 (+ 2645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:16:36.248191 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:16:36.250191 (+ 2000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:16:36.250201 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:16:36.250397 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:36.250486 (+ 89us) client_negotiation.cc:769] Sending connection context
0620 20:16:36.250558 (+ 72us) client_negotiation.cc:240] Negotiation successful
0620 20:16:36.250787 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11919,"mutex_wait_us":70,"thread_start_us":23577,"threads_started":1}
I20260620 20:16:36.252636 24709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:36.222631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53902 (local address 127.16.217.3:46165)
0620 20:16:36.240805 (+ 18174us) server_negotiation.cc:206] Beginning negotiation
0620 20:16:36.240828 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0620 20:16:36.240874 (+ 46us) server_negotiation.cc:407] Connection header received
0620 20:16:36.241009 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:16:36.241021 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:16:36.241385 (+ 364us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:16:36.242625 (+ 1240us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:16:36.243072 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:36.245320 (+ 2248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:36.248338 (+ 3018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:36.249948 (+ 1610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:36.251835 (+ 1887us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:36.252065 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0620 20:16:36.252139 (+ 74us) server_negotiation.cc:299] Negotiation successful
0620 20:16:36.252304 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17949,"thread_start_us":17756,"threads_started":1}
I20260620 20:16:36.254935 24705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:36.221715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.1:34627 (local address 127.0.0.1:35954)
0620 20:16:36.233938 (+ 12223us) negotiation.cc:107] Waiting for socket to connect
0620 20:16:36.233964 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0620 20:16:36.234136 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:16:36.246100 (+ 11964us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:16:36.246106 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:16:36.246134 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:16:36.246802 (+ 668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:16:36.246818 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:16:36.249951 (+ 3133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:16:36.249956 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:16:36.252234 (+ 2278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:16:36.252247 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:16:36.254121 (+ 1874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:16:36.254128 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:16:36.254286 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:36.254318 (+ 32us) client_negotiation.cc:769] Sending connection context
0620 20:16:36.254384 (+ 66us) client_negotiation.cc:240] Negotiation successful
0620 20:16:36.254499 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12009,"mutex_wait_us":1,"thread_start_us":620,"threads_started":1}
I20260620 20:16:36.255764 24707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:36.222668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.2:41479 (local address 127.0.0.1:48680)
0620 20:16:36.247216 (+ 24548us) negotiation.cc:107] Waiting for socket to connect
0620 20:16:36.247240 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 20:16:36.247366 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:16:36.249151 (+ 1785us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:16:36.249156 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:16:36.249179 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:16:36.249643 (+ 464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:16:36.249656 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:16:36.251493 (+ 1837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:16:36.251498 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:16:36.253560 (+ 2062us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:16:36.253576 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:16:36.254882 (+ 1306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:16:36.254892 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:16:36.255160 (+ 268us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:36.255209 (+ 49us) client_negotiation.cc:769] Sending connection context
0620 20:16:36.255306 (+ 97us) client_negotiation.cc:240] Negotiation successful
0620 20:16:36.255470 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24431,"thread_start_us":11279,"threads_started":1}
I20260620 20:16:36.256124 24706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:36.221984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35954 (local address 127.16.217.1:34627)
0620 20:16:36.245043 (+ 23059us) server_negotiation.cc:206] Beginning negotiation
0620 20:16:36.245051 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 20:16:36.245080 (+ 29us) server_negotiation.cc:407] Connection header received
0620 20:16:36.245172 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:16:36.245180 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:16:36.245429 (+ 249us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:16:36.246215 (+ 786us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:16:36.246990 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:36.249781 (+ 2791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:36.252460 (+ 2679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:36.253909 (+ 1449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:36.255265 (+ 1356us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:36.255480 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0620 20:16:36.255547 (+ 67us) server_negotiation.cc:299] Negotiation successful
0620 20:16:36.255718 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22889,"thread_start_us":23793,"threads_started":1}
I20260620 20:16:36.257423 24710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:36.223016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48680 (local address 127.16.217.2:41479)
0620 20:16:36.239929 (+ 16913us) server_negotiation.cc:206] Beginning negotiation
0620 20:16:36.239943 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 20:16:36.248366 (+ 8423us) server_negotiation.cc:407] Connection header received
0620 20:16:36.248473 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:16:36.248480 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:16:36.248732 (+ 252us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:16:36.248932 (+ 200us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:16:36.249833 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:36.251332 (+ 1499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:36.253733 (+ 2401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:36.254663 (+ 930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:36.254821 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:36.255021 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0620 20:16:36.255359 (+ 338us) server_negotiation.cc:299] Negotiation successful
0620 20:16:36.256069 (+ 710us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16651,"thread_start_us":16934,"threads_started":1}
I20260620 20:16:36.257368 24610 tablet_service.cc:1467] Processing CreateTablet for tablet 8f501dd30f5b4b80acfa05993ee087c8 (DEFAULT_TABLE table=test-table [id=aa6473fc517d4c04819cfab01d3bcf74]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:16:36.259064 24321 tablet_service.cc:1467] Processing CreateTablet for tablet 8f501dd30f5b4b80acfa05993ee087c8 (DEFAULT_TABLE table=test-table [id=aa6473fc517d4c04819cfab01d3bcf74]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:16:36.260241 24610 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f501dd30f5b4b80acfa05993ee087c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:16:36.260243 24321 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f501dd30f5b4b80acfa05993ee087c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:16:36.260201 24468 tablet_service.cc:1467] Processing CreateTablet for tablet 8f501dd30f5b4b80acfa05993ee087c8 (DEFAULT_TABLE table=test-table [id=aa6473fc517d4c04819cfab01d3bcf74]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:16:36.261451 24468 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f501dd30f5b4b80acfa05993ee087c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:16:36.273494 24711 tablet_bootstrap.cc:492] T 8f501dd30f5b4b80acfa05993ee087c8 P 33fb52c1b00347a8ac5db344e2959cac: Bootstrap starting.
I20260620 20:16:36.275413 24713 tablet_bootstrap.cc:492] T 8f501dd30f5b4b80acfa05993ee087c8 P b6d268e1e2e944af9e30e4b3e4b3e359: Bootstrap starting.
I20260620 20:16:36.276788 24712 tablet_bootstrap.cc:492] T 8f501dd30f5b4b80acfa05993ee087c8 P 5280c2f4f3ef4c77890e5b5b7c7f78ae: Bootstrap starting.
I20260620 20:16:36.278985 24711 tablet_bootstrap.cc:654] T 8f501dd30f5b4b80acfa05993ee087c8 P 33fb52c1b00347a8ac5db344e2959cac: Neither blocks nor log segments found. Creating new log.
I20260620 20:16:36.279170 24713 tablet_bootstrap.cc:654] T 8f501dd30f5b4b80acfa05993ee087c8 P b6d268e1e2e944af9e30e4b3e4b3e359: Neither blocks nor log segments found. Creating new log.
I20260620 20:16:36.279522 24712 tablet_bootstrap.cc:654] T 8f501dd30f5b4b80acfa05993ee087c8 P 5280c2f4f3ef4c77890e5b5b7c7f78ae: Neither blocks nor log segments found. Creating new log.
I20260620 20:16:36.280294 24713 log.cc:824] T 8f501dd30f5b4b80acfa05993ee087c8 P b6d268e1e2e944af9e30e4b3e4b3e359: Log is configured to *not* fsync() on all Append() calls
I20260620 20:16:36.280707 24712 log.cc:824] T 8f501dd30f5b4b80acfa05993ee087c8 P 5280c2f4f3ef4c77890e5b5b7c7f78ae: Log is configured to *not* fsync() on all Append() calls
I20260620 20:16:36.280493 24711 log.cc:824] T 8f501dd30f5b4b80acfa05993ee087c8 P 33fb52c1b00347a8ac5db344e2959cac: Log is configured to *not* fsync() on all Append() calls
I20260620 20:16:36.283492 24713 tablet_bootstrap.cc:492] T 8f501dd30f5b4b80acfa05993ee087c8 P b6d268e1e2e944af9e30e4b3e4b3e359: No bootstrap required, opened a new log
I20260620 20:16:36.283774 24713 ts_tablet_manager.cc:1397] T 8f501dd30f5b4b80acfa05993ee087c8 P b6d268e1e2e944af9e30e4b3e4b3e359: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.005s
I20260620 20:16:36.284375 24712 tablet_bootstrap.cc:492] T 8f501dd30f5b4b80acfa05993ee087c8 P 5280c2f4f3ef4c77890e5b5b7c7f78ae: No bootstrap required, opened a new log
I20260620 20:16:36.284618 24712 ts_tablet_manager.cc:1397] T 8f501dd30f5b4b80acfa05993ee087c8 P 5280c2f4f3ef4c77890e5b5b7c7f78ae: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.005s
I20260620 20:16:36.287889 24711 tablet_bootstrap.cc:492] T 8f501dd30f5b4b80acfa05993ee087c8 P 33fb52c1b00347a8ac5db344e2959cac: No bootstrap required, opened a new log
I20260620 20:16:36.288148 24711 ts_tablet_manager.cc:1397] T 8f501dd30f5b4b80acfa05993ee087c8 P 33fb52c1b00347a8ac5db344e2959cac: Time spent bootstrapping tablet: real 0.015s user 0.007s sys 0.004s
I20260620 20:16:36.294502 24713 raft_consensus.cc:348] T 8f501dd30f5b4b80acfa05993ee087c8 P b6d268e1e2e944af9e30e4b3e4b3e359 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5280c2f4f3ef4c77890e5b5b7c7f78ae" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 34627 } } peers { permanent_uuid: "b6d268e1e2e944af9e30e4b3e4b3e359" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 46165 } } peers { permanent_uuid: "33fb52c1b00347a8ac5db344e2959cac" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 41479 } }
I20260620 20:16:36.295007 24713 raft_consensus.cc:374] T 8f501dd30f5b4b80acfa05993ee087c8 P b6d268e1e2e944af9e30e4b3e4b3e359 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:16:36.295117 24713 raft_consensus.cc:729] T 8f501dd30f5b4b80acfa05993ee087c8 P b6d268e1e2e944af9e30e4b3e4b3e359 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6d268e1e2e944af9e30e4b3e4b3e359, State: Initialized, Role: FOLLOWER
I20260620 20:16:36.295054 24712 raft_consensus.cc:348] T 8f501dd30f5b4b80acfa05993ee087c8 P 5280c2f4f3ef4c77890e5b5b7c7f78ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5280c2f4f3ef4c77890e5b5b7c7f78ae" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 34627 } } peers { permanent_uuid: "b6d268e1e2e944af9e30e4b3e4b3e359" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 46165 } } peers { permanent_uuid: "33fb52c1b00347a8ac5db344e2959cac" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 41479 } }
I20260620 20:16:36.295473 24712 raft_consensus.cc:374] T 8f501dd30f5b4b80acfa05993ee087c8 P 5280c2f4f3ef4c77890e5b5b7c7f78ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:16:36.295567 24712 raft_consensus.cc:729] T 8f501dd30f5b4b80acfa05993ee087c8 P 5280c2f4f3ef4c77890e5b5b7c7f78ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5280c2f4f3ef4c77890e5b5b7c7f78ae, State: Initialized, Role: FOLLOWER
I20260620 20:16:36.295388 24711 raft_consensus.cc:348] T 8f501dd30f5b4b80acfa05993ee087c8 P 33fb52c1b00347a8ac5db344e2959cac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5280c2f4f3ef4c77890e5b5b7c7f78ae" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 34627 } } peers { permanent_uuid: "b6d268e1e2e944af9e30e4b3e4b3e359" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 46165 } } peers { permanent_uuid: "33fb52c1b00347a8ac5db344e2959cac" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 41479 } }
I20260620 20:16:36.295828 24711 raft_consensus.cc:374] T 8f501dd30f5b4b80acfa05993ee087c8 P 33fb52c1b00347a8ac5db344e2959cac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:16:36.295789 24713 consensus_queue.cc:260] T 8f501dd30f5b4b80acfa05993ee087c8 P b6d268e1e2e944af9e30e4b3e4b3e359 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5280c2f4f3ef4c77890e5b5b7c7f78ae" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 34627 } } peers { permanent_uuid: "b6d268e1e2e944af9e30e4b3e4b3e359" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 46165 } } peers { permanent_uuid: "33fb52c1b00347a8ac5db344e2959cac" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 41479 } }
I20260620 20:16:36.296216 24712 consensus_queue.cc:260] T 8f501dd30f5b4b80acfa05993ee087c8 P 5280c2f4f3ef4c77890e5b5b7c7f78ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5280c2f4f3ef4c77890e5b5b7c7f78ae" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 34627 } } peers { permanent_uuid: "b6d268e1e2e944af9e30e4b3e4b3e359" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 46165 } } peers { permanent_uuid: "33fb52c1b00347a8ac5db344e2959cac" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 41479 } }
I20260620 20:16:36.297492 24711 raft_consensus.cc:729] T 8f501dd30f5b4b80acfa05993ee087c8 P 33fb52c1b00347a8ac5db344e2959cac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33fb52c1b00347a8ac5db344e2959cac, State: Initialized, Role: FOLLOWER
I20260620 20:16:36.298234 24711 consensus_queue.cc:260] T 8f501dd30f5b4b80acfa05993ee087c8 P 33fb52c1b00347a8ac5db344e2959cac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5280c2f4f3ef4c77890e5b5b7c7f78ae" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 34627 } } peers { permanent_uuid: "b6d268e1e2e944af9e30e4b3e4b3e359" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 46165 } } peers { permanent_uuid: "33fb52c1b00347a8ac5db344e2959cac" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 41479 } }
I20260620 20:16:36.298451 24713 ts_tablet_manager.cc:1428] T 8f501dd30f5b4b80acfa05993ee087c8 P b6d268e1e2e944af9e30e4b3e4b3e359: Time spent starting tablet: real 0.015s user 0.011s sys 0.005s
I20260620 20:16:36.298787 24676 heartbeater.cc:503] Master 127.16.217.62:36319 was elected leader, sending a full tablet report...
I20260620 20:16:36.304517 24534 heartbeater.cc:503] Master 127.16.217.62:36319 was elected leader, sending a full tablet report...
I20260620 20:16:36.305204 24711 ts_tablet_manager.cc:1428] T 8f501dd30f5b4b80acfa05993ee087c8 P 33fb52c1b00347a8ac5db344e2959cac: Time spent starting tablet: real 0.017s user 0.011s sys 0.000s
I20260620 20:16:36.308925 24712 ts_tablet_manager.cc:1428] T 8f501dd30f5b4b80acfa05993ee087c8 P 5280c2f4f3ef4c77890e5b5b7c7f78ae: Time spent starting tablet: real 0.024s user 0.013s sys 0.009s
W20260620 20:16:36.390467 24678 tablet.cc:2369] T 8f501dd30f5b4b80acfa05993ee087c8 P b6d268e1e2e944af9e30e4b3e4b3e359: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 20:16:36.427790 24389 tablet.cc:2369] T 8f501dd30f5b4b80acfa05993ee087c8 P 5280c2f4f3ef4c77890e5b5b7c7f78ae: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 20:16:36.521494 24536 tablet.cc:2369] T 8f501dd30f5b4b80acfa05993ee087c8 P 33fb52c1b00347a8ac5db344e2959cac: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 20:16:36.521759 24718 raft_consensus.cc:482] T 8f501dd30f5b4b80acfa05993ee087c8 P b6d268e1e2e944af9e30e4b3e4b3e359 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 20:16:36.522112 24718 raft_consensus.cc:504] T 8f501dd30f5b4b80acfa05993ee087c8 P b6d268e1e2e944af9e30e4b3e4b3e359 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5280c2f4f3ef4c77890e5b5b7c7f78ae" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 34627 } } peers { permanent_uuid: "b6d268e1e2e944af9e30e4b3e4b3e359" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 46165 } } peers { permanent_uuid: "33fb52c1b00347a8ac5db344e2959cac" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 41479 } }
I20260620 20:16:36.523943 24718 leader_election.cc:288] T 8f501dd30f5b4b80acfa05993ee087c8 P b6d268e1e2e944af9e30e4b3e4b3e359 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5280c2f4f3ef4c77890e5b5b7c7f78ae (127.16.217.1:34627), 33fb52c1b00347a8ac5db344e2959cac (127.16.217.2:41479)
I20260620 20:16:36.530258 24706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:36.524322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.217.3:45661 (local address 127.16.217.1:34627)
0620 20:16:36.524544 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0620 20:16:36.524550 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:16:36.524573 (+ 23us) server_negotiation.cc:407] Connection header received
0620 20:16:36.524738 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:16:36.524745 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:16:36.524870 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:16:36.524980 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:16:36.525351 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:36.526150 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:36.528102 (+ 1952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:36.529118 (+ 1016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:36.529274 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:36.529328 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0620 20:16:36.529752 (+ 424us) server_negotiation.cc:299] Negotiation successful
0620 20:16:36.529912 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 20:16:36.530344 24680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:36.524182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.1:34627 (local address 127.16.217.3:45661)
0620 20:16:36.524379 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0620 20:16:36.524403 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 20:16:36.524580 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:16:36.524996 (+ 416us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:16:36.525000 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:16:36.525024 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:16:36.525210 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:16:36.525218 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:16:36.526302 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:16:36.526308 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:16:36.527949 (+ 1641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:16:36.527962 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:16:36.529290 (+ 1328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:16:36.529297 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:16:36.529565 (+ 268us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:36.529608 (+ 43us) client_negotiation.cc:769] Sending connection context
0620 20:16:36.529713 (+ 105us) client_negotiation.cc:240] Negotiation successful
0620 20:16:36.529903 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":66}
I20260620 20:16:36.530404 24685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:36.524543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.2:41479 (local address 127.16.217.3:55697)
0620 20:16:36.524705 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0620 20:16:36.524724 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 20:16:36.524839 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:16:36.525210 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:16:36.525215 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:16:36.525239 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:16:36.525421 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:16:36.525430 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:16:36.526719 (+ 1289us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:16:36.526741 (+ 22us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:16:36.528257 (+ 1516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:16:36.528271 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:16:36.529631 (+ 1360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:16:36.529636 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:16:36.529835 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:36.529884 (+ 49us) client_negotiation.cc:769] Sending connection context
0620 20:16:36.529967 (+ 83us) client_negotiation.cc:240] Negotiation successful
0620 20:16:36.530094 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53}
I20260620 20:16:36.530926 24710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:36.524721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.217.3:55697 (local address 127.16.217.2:41479)
0620 20:16:36.524930 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0620 20:16:36.524937 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:16:36.524964 (+ 27us) server_negotiation.cc:407] Connection header received
0620 20:16:36.525019 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:16:36.525022 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:16:36.525078 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:16:36.525185 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:16:36.525572 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:36.526582 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:36.528413 (+ 1831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:36.529476 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:36.529654 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:36.529756 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0620 20:16:36.530602 (+ 846us) server_negotiation.cc:299] Negotiation successful
0620 20:16:36.530749 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 20:16:36.531499 24341 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f501dd30f5b4b80acfa05993ee087c8" candidate_uuid: "b6d268e1e2e944af9e30e4b3e4b3e359" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5280c2f4f3ef4c77890e5b5b7c7f78ae" is_pre_election: true
I20260620 20:16:36.531983 24341 raft_consensus.cc:2454] T 8f501dd30f5b4b80acfa05993ee087c8 P 5280c2f4f3ef4c77890e5b5b7c7f78ae [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b6d268e1e2e944af9e30e4b3e4b3e359 in term 0.
I20260620 20:16:36.532701 24565 leader_election.cc:302] T 8f501dd30f5b4b80acfa05993ee087c8 P b6d268e1e2e944af9e30e4b3e4b3e359 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5280c2f4f3ef4c77890e5b5b7c7f78ae, b6d268e1e2e944af9e30e4b3e4b3e359; no voters:
I20260620 20:16:36.533136 24718 raft_consensus.cc:2781] T 8f501dd30f5b4b80acfa05993ee087c8 P b6d268e1e2e944af9e30e4b3e4b3e359 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 20:16:36.533032 24488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f501dd30f5b4b80acfa05993ee087c8" candidate_uuid: "b6d268e1e2e944af9e30e4b3e4b3e359" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33fb52c1b00347a8ac5db344e2959cac" is_pre_election: true
I20260620 20:16:36.533249 24718 raft_consensus.cc:482] T 8f501dd30f5b4b80acfa05993ee087c8 P b6d268e1e2e944af9e30e4b3e4b3e359 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 20:16:36.533396 24718 raft_consensus.cc:3037] T 8f501dd30f5b4b80acfa05993ee087c8 P b6d268e1e2e944af9e30e4b3e4b3e359 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:16:36.533528 24488 raft_consensus.cc:2454] T 8f501dd30f5b4b80acfa05993ee087c8 P 33fb52c1b00347a8ac5db344e2959cac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b6d268e1e2e944af9e30e4b3e4b3e359 in term 0.
I20260620 20:16:36.536326 24718 raft_consensus.cc:504] T 8f501dd30f5b4b80acfa05993ee087c8 P b6d268e1e2e944af9e30e4b3e4b3e359 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5280c2f4f3ef4c77890e5b5b7c7f78ae" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 34627 } } peers { permanent_uuid: "b6d268e1e2e944af9e30e4b3e4b3e359" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 46165 } } peers { permanent_uuid: "33fb52c1b00347a8ac5db344e2959cac" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 41479 } }
I20260620 20:16:36.536939 24718 leader_election.cc:288] T 8f501dd30f5b4b80acfa05993ee087c8 P b6d268e1e2e944af9e30e4b3e4b3e359 [CANDIDATE]: Term 1 election: Requested vote from peers 5280c2f4f3ef4c77890e5b5b7c7f78ae (127.16.217.1:34627), 33fb52c1b00347a8ac5db344e2959cac (127.16.217.2:41479)
I20260620 20:16:36.537387 24341 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f501dd30f5b4b80acfa05993ee087c8" candidate_uuid: "b6d268e1e2e944af9e30e4b3e4b3e359" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5280c2f4f3ef4c77890e5b5b7c7f78ae"
I20260620 20:16:36.537449 24488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f501dd30f5b4b80acfa05993ee087c8" candidate_uuid: "b6d268e1e2e944af9e30e4b3e4b3e359" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33fb52c1b00347a8ac5db344e2959cac"
I20260620 20:16:36.537604 24341 raft_consensus.cc:3037] T 8f501dd30f5b4b80acfa05993ee087c8 P 5280c2f4f3ef4c77890e5b5b7c7f78ae [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:16:36.537657 24488 raft_consensus.cc:3037] T 8f501dd30f5b4b80acfa05993ee087c8 P 33fb52c1b00347a8ac5db344e2959cac [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:16:36.540872 24341 raft_consensus.cc:2454] T 8f501dd30f5b4b80acfa05993ee087c8 P 5280c2f4f3ef4c77890e5b5b7c7f78ae [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b6d268e1e2e944af9e30e4b3e4b3e359 in term 1.
I20260620 20:16:36.540865 24488 raft_consensus.cc:2454] T 8f501dd30f5b4b80acfa05993ee087c8 P 33fb52c1b00347a8ac5db344e2959cac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b6d268e1e2e944af9e30e4b3e4b3e359 in term 1.
I20260620 20:16:36.541237 24565 leader_election.cc:302] T 8f501dd30f5b4b80acfa05993ee087c8 P b6d268e1e2e944af9e30e4b3e4b3e359 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5280c2f4f3ef4c77890e5b5b7c7f78ae, b6d268e1e2e944af9e30e4b3e4b3e359; no voters:
I20260620 20:16:36.541569 24718 raft_consensus.cc:2781] T 8f501dd30f5b4b80acfa05993ee087c8 P b6d268e1e2e944af9e30e4b3e4b3e359 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:16:36.542019 24718 raft_consensus.cc:686] T 8f501dd30f5b4b80acfa05993ee087c8 P b6d268e1e2e944af9e30e4b3e4b3e359 [term 1 LEADER]: Becoming Leader. State: Replica: b6d268e1e2e944af9e30e4b3e4b3e359, State: Running, Role: LEADER
I20260620 20:16:36.542572 24718 consensus_queue.cc:237] T 8f501dd30f5b4b80acfa05993ee087c8 P b6d268e1e2e944af9e30e4b3e4b3e359 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5280c2f4f3ef4c77890e5b5b7c7f78ae" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 34627 } } peers { permanent_uuid: "b6d268e1e2e944af9e30e4b3e4b3e359" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 46165 } } peers { permanent_uuid: "33fb52c1b00347a8ac5db344e2959cac" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 41479 } }
I20260620 20:16:36.547542 23414 catalog_manager.cc:5495] T 8f501dd30f5b4b80acfa05993ee087c8 P b6d268e1e2e944af9e30e4b3e4b3e359 reported cstate change: term changed from 0 to 1, leader changed from <none> to b6d268e1e2e944af9e30e4b3e4b3e359 (127.16.217.3). New cstate: current_term: 1 leader_uuid: "b6d268e1e2e944af9e30e4b3e4b3e359" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5280c2f4f3ef4c77890e5b5b7c7f78ae" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 34627 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b6d268e1e2e944af9e30e4b3e4b3e359" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 46165 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "33fb52c1b00347a8ac5db344e2959cac" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 41479 } health_report { overall_health: UNKNOWN } } }
I20260620 20:16:36.595028 24709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:36.590144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53908 (local address 127.16.217.3:46165)
0620 20:16:36.590327 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0620 20:16:36.590332 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:16:36.590349 (+ 17us) server_negotiation.cc:407] Connection header received
0620 20:16:36.590544 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:16:36.590547 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:16:36.590621 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:16:36.590751 (+ 130us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 20:16:36.591062 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:36.591769 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:36.593047 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:36.593467 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:36.593568 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:36.594054 (+ 486us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 20:16:36.594561 (+ 507us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 20:16:36.594675 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0620 20:16:36.594765 (+ 90us) server_negotiation.cc:299] Negotiation successful
0620 20:16:36.594826 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 20:16:36.603192 24488 raft_consensus.cc:1261] T 8f501dd30f5b4b80acfa05993ee087c8 P 33fb52c1b00347a8ac5db344e2959cac [term 1 FOLLOWER]: Refusing update from remote peer b6d268e1e2e944af9e30e4b3e4b3e359: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:16:36.603192 24341 raft_consensus.cc:1261] T 8f501dd30f5b4b80acfa05993ee087c8 P 5280c2f4f3ef4c77890e5b5b7c7f78ae [term 1 FOLLOWER]: Refusing update from remote peer b6d268e1e2e944af9e30e4b3e4b3e359: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:16:36.604187 24721 consensus_queue.cc:1035] T 8f501dd30f5b4b80acfa05993ee087c8 P b6d268e1e2e944af9e30e4b3e4b3e359 [LEADER]: Connected to new peer: Peer: permanent_uuid: "33fb52c1b00347a8ac5db344e2959cac" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 41479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:16:36.604559 24718 consensus_queue.cc:1035] T 8f501dd30f5b4b80acfa05993ee087c8 P b6d268e1e2e944af9e30e4b3e4b3e359 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5280c2f4f3ef4c77890e5b5b7c7f78ae" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 34627 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:16:36.638813 23415 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50326:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 20:16:36.639066 23415 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50326:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 20:16:36.646685 23415 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 358753b177ca4441ba3b33b1e5fd7d66: Sending DeleteTablet for 3 replicas of tablet 8f501dd30f5b4b80acfa05993ee087c8
I20260620 20:16:36.647971 24321 tablet_service.cc:1514] Processing DeleteTablet for tablet 8f501dd30f5b4b80acfa05993ee087c8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:16:36 UTC) from {username='oryx', principal='oryx/127.16.217.62@KRBTEST.COM'} at 127.0.0.1:35954
I20260620 20:16:36.648432 24468 tablet_service.cc:1514] Processing DeleteTablet for tablet 8f501dd30f5b4b80acfa05993ee087c8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:16:36 UTC) from {username='oryx', principal='oryx/127.16.217.62@KRBTEST.COM'} at 127.0.0.1:48680
I20260620 20:16:36.648618 24610 tablet_service.cc:1514] Processing DeleteTablet for tablet 8f501dd30f5b4b80acfa05993ee087c8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:16:36 UTC) from {username='oryx', principal='oryx/127.16.217.62@KRBTEST.COM'} at 127.0.0.1:53902
I20260620 20:16:36.648845 24735 tablet_replica.cc:330] T 8f501dd30f5b4b80acfa05993ee087c8 P 5280c2f4f3ef4c77890e5b5b7c7f78ae: stopping tablet replica
I20260620 20:16:36.649554 24735 raft_consensus.cc:2229] T 8f501dd30f5b4b80acfa05993ee087c8 P 5280c2f4f3ef4c77890e5b5b7c7f78ae [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 20:16:36.650096 24735 raft_consensus.cc:2258] T 8f501dd30f5b4b80acfa05993ee087c8 P 5280c2f4f3ef4c77890e5b5b7c7f78ae [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:16:36.651055 24737 tablet_replica.cc:330] T 8f501dd30f5b4b80acfa05993ee087c8 P b6d268e1e2e944af9e30e4b3e4b3e359: stopping tablet replica
I20260620 20:16:36.651450 24737 raft_consensus.cc:2229] T 8f501dd30f5b4b80acfa05993ee087c8 P b6d268e1e2e944af9e30e4b3e4b3e359 [term 1 LEADER]: Raft consensus shutting down.
I20260620 20:16:36.652063 24737 raft_consensus.cc:2258] T 8f501dd30f5b4b80acfa05993ee087c8 P b6d268e1e2e944af9e30e4b3e4b3e359 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:16:36.652722 24735 ts_tablet_manager.cc:1905] T 8f501dd30f5b4b80acfa05993ee087c8 P 5280c2f4f3ef4c77890e5b5b7c7f78ae: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:16:36.652745 24736 tablet_replica.cc:330] T 8f501dd30f5b4b80acfa05993ee087c8 P 33fb52c1b00347a8ac5db344e2959cac: stopping tablet replica
I20260620 20:16:36.653127 24736 raft_consensus.cc:2229] T 8f501dd30f5b4b80acfa05993ee087c8 P 33fb52c1b00347a8ac5db344e2959cac [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 20:16:36.653607 24736 raft_consensus.cc:2258] T 8f501dd30f5b4b80acfa05993ee087c8 P 33fb52c1b00347a8ac5db344e2959cac [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:16:36.654050 24737 ts_tablet_manager.cc:1905] T 8f501dd30f5b4b80acfa05993ee087c8 P b6d268e1e2e944af9e30e4b3e4b3e359: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:16:36.661682 24737 ts_tablet_manager.cc:1918] T 8f501dd30f5b4b80acfa05993ee087c8 P b6d268e1e2e944af9e30e4b3e4b3e359: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 20:16:36.661892 24737 log.cc:1196] T 8f501dd30f5b4b80acfa05993ee087c8 P b6d268e1e2e944af9e30e4b3e4b3e359: Deleting WAL directory at /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986536687152-17252-0/minicluster-data/ts-2/wal/wals/8f501dd30f5b4b80acfa05993ee087c8
I20260620 20:16:36.661834 24736 ts_tablet_manager.cc:1905] T 8f501dd30f5b4b80acfa05993ee087c8 P 33fb52c1b00347a8ac5db344e2959cac: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:16:36.662488 24737 ts_tablet_manager.cc:1939] T 8f501dd30f5b4b80acfa05993ee087c8 P b6d268e1e2e944af9e30e4b3e4b3e359: Deleting consensus metadata
I20260620 20:16:36.664445 23401 catalog_manager.cc:4841] TS b6d268e1e2e944af9e30e4b3e4b3e359 (127.16.217.3:46165): tablet 8f501dd30f5b4b80acfa05993ee087c8 (table test-table [id=aa6473fc517d4c04819cfab01d3bcf74]) successfully deleted
I20260620 20:16:36.665360 17252 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu pid 23383
I20260620 20:16:36.666460 23445 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:16:36.668622 24735 ts_tablet_manager.cc:1918] T 8f501dd30f5b4b80acfa05993ee087c8 P 5280c2f4f3ef4c77890e5b5b7c7f78ae: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 20:16:36.668835 24735 log.cc:1196] T 8f501dd30f5b4b80acfa05993ee087c8 P 5280c2f4f3ef4c77890e5b5b7c7f78ae: Deleting WAL directory at /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986536687152-17252-0/minicluster-data/ts-0/wal/wals/8f501dd30f5b4b80acfa05993ee087c8
I20260620 20:16:36.669548 24735 ts_tablet_manager.cc:1939] T 8f501dd30f5b4b80acfa05993ee087c8 P 5280c2f4f3ef4c77890e5b5b7c7f78ae: Deleting consensus metadata
I20260620 20:16:36.674924 24736 ts_tablet_manager.cc:1918] T 8f501dd30f5b4b80acfa05993ee087c8 P 33fb52c1b00347a8ac5db344e2959cac: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 20:16:36.677824 24736 log.cc:1196] T 8f501dd30f5b4b80acfa05993ee087c8 P 33fb52c1b00347a8ac5db344e2959cac: Deleting WAL directory at /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986536687152-17252-0/minicluster-data/ts-1/wal/wals/8f501dd30f5b4b80acfa05993ee087c8
I20260620 20:16:36.678568 24736 ts_tablet_manager.cc:1939] T 8f501dd30f5b4b80acfa05993ee087c8 P 33fb52c1b00347a8ac5db344e2959cac: Deleting consensus metadata
I20260620 20:16:36.687194 24689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:36.651648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59168 (local address 127.16.217.61:33441)
0620 20:16:36.651832 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0620 20:16:36.651838 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:16:36.663630 (+ 11792us) server_negotiation.cc:407] Connection header received
0620 20:16:36.664980 (+ 1350us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:16:36.664986 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:16:36.665066 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:16:36.665133 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:16:36.671827 (+ 6694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:36.672968 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:36.675091 (+ 2123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:36.675933 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:36.676044 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:36.679955 (+ 3911us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:16:36.679962 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:16:36.679969 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:16:36.680010 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:16:36.683802 (+ 3792us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:16:36.684507 (+ 705us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:16:36.684513 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:16:36.684517 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:16:36.684645 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:16:36.685282 (+ 637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:16:36.685673 (+ 391us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:16:36.685676 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:16:36.686070 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:16:36.686326 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0620 20:16:36.686624 (+ 298us) server_negotiation.cc:299] Negotiation successful
0620 20:16:36.686838 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 20:16:36.706239 24681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:36.664164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48972 (local address 127.16.217.60:37035)
0620 20:16:36.664352 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0620 20:16:36.664358 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:16:36.687343 (+ 22985us) server_negotiation.cc:407] Connection header received
0620 20:16:36.687598 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:16:36.687603 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:16:36.687689 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:16:36.687755 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:16:36.689423 (+ 1668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:36.690372 (+ 949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:36.695674 (+ 5302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:36.696315 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:36.696406 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:36.699839 (+ 3433us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:16:36.699846 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:16:36.699852 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:16:36.699895 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:16:36.703226 (+ 3331us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:16:36.704022 (+ 796us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:16:36.704028 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:16:36.704032 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:16:36.704164 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:16:36.704556 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:16:36.704561 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:16:36.704565 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:16:36.704901 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:16:36.705120 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0620 20:16:36.705639 (+ 519us) server_negotiation.cc:299] Negotiation successful
0620 20:16:36.705859 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 20:16:36.835399 23402 catalog_manager.cc:4841] TS 5280c2f4f3ef4c77890e5b5b7c7f78ae (127.16.217.1:34627): tablet 8f501dd30f5b4b80acfa05993ee087c8 (table test-table [id=aa6473fc517d4c04819cfab01d3bcf74]) successfully deleted
I20260620 20:16:36.835602 17252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu with pid 23383
W20260620 20:16:36.853317 24420 connection.cc:535] server connection from 127.0.0.1:48680 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 20:16:36.876931 24681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:36.857833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48986 (local address 127.16.217.60:37035)
0620 20:16:36.858041 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0620 20:16:36.858047 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:16:36.861417 (+ 3370us) server_negotiation.cc:407] Connection header received
0620 20:16:36.861659 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:16:36.861666 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:16:36.861758 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:16:36.861866 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:16:36.863391 (+ 1525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:36.864183 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:36.866043 (+ 1860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:36.866677 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:36.866817 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:36.869646 (+ 2829us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:16:36.869651 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:16:36.869654 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:16:36.869686 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:16:36.873566 (+ 3880us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:16:36.874409 (+ 843us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:16:36.874418 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:16:36.874424 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:16:36.874586 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:16:36.875070 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:16:36.875076 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:16:36.875080 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:16:36.875494 (+ 414us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:16:36.875743 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0620 20:16:36.876361 (+ 618us) server_negotiation.cc:299] Negotiation successful
0620 20:16:36.876600 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260620 20:16:36.877756 24689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:36.857805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59176 (local address 127.16.217.61:33441)
0620 20:16:36.858016 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0620 20:16:36.858022 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:16:36.861074 (+ 3052us) server_negotiation.cc:407] Connection header received
0620 20:16:36.861760 (+ 686us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:16:36.861766 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:16:36.861846 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:16:36.861965 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:16:36.864173 (+ 2208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:36.864942 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:36.866582 (+ 1640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:36.867203 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:36.867321 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:36.870695 (+ 3374us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:16:36.870704 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:16:36.870710 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:16:36.870753 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:16:36.874516 (+ 3763us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:16:36.875305 (+ 789us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:16:36.875311 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:16:36.875314 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:16:36.875461 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:16:36.875946 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:16:36.875954 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:16:36.875960 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:16:36.876335 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:16:36.876562 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0620 20:16:36.877127 (+ 565us) server_negotiation.cc:299] Negotiation successful
0620 20:16:36.877388 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260620 20:16:37.636785 24747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:37.634528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:36319 (local address 127.16.217.3:59221)
0620 20:16:37.636338 (+ 1810us) negotiation.cc:107] Waiting for socket to connect
0620 20:16:37.636483 (+ 145us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.217.62:36319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1605,"thread_start_us":1503,"threads_started":1}
I20260620 20:16:37.636940 24748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:37.635159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:36319 (local address 127.16.217.2:36949)
0620 20:16:37.636599 (+ 1440us) negotiation.cc:107] Waiting for socket to connect
0620 20:16:37.636727 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.217.62:36319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1338,"thread_start_us":1238,"threads_started":1}
W20260620 20:16:37.637303 24676 heartbeater.cc:644] Failed to heartbeat to 127.16.217.62:36319 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.16.217.62:36319: connect: Connection refused (error 111)
W20260620 20:16:37.637336 24534 heartbeater.cc:644] Failed to heartbeat to 127.16.217.62:36319 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.16.217.62:36319: connect: Connection refused (error 111)
I20260620 20:16:37.637809 24749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:37.636537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:36319 (local address 127.16.217.1:41923)
0620 20:16:37.637315 (+ 778us) negotiation.cc:107] Waiting for socket to connect
0620 20:16:37.637455 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.217.62:36319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":665,"threads_started":1}
W20260620 20:16:37.638340 24387 heartbeater.cc:644] Failed to heartbeat to 127.16.217.62:36319 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.16.217.62:36319: connect: Connection refused (error 111)
I20260620 20:16:38.145545 24750 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5c48720af35c483da74e1243b72c87c3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 358753b177ca4441ba3b33b1e5fd7d66)
I20260620 20:16:38.145937 24750 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5c48720af35c483da74e1243b72c87c3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "358753b177ca4441ba3b33b1e5fd7d66" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 36319 } } peers { permanent_uuid: "0fe54b7d8d1b498f81774ec398b1f6aa" member_type: VOTER last_known_addr { host: "127.16.217.61" port: 33441 } attrs { promote: false } } peers { permanent_uuid: "5c48720af35c483da74e1243b72c87c3" member_type: VOTER last_known_addr { host: "127.16.217.60" port: 37035 } attrs { promote: false } }
I20260620 20:16:38.147439 24750 leader_election.cc:288] T 00000000000000000000000000000000 P 5c48720af35c483da74e1243b72c87c3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 358753b177ca4441ba3b33b1e5fd7d66 (127.16.217.62:36319), 0fe54b7d8d1b498f81774ec398b1f6aa (127.16.217.61:33441)
I20260620 20:16:38.149127 24751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:38.147621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:36319 (local address 127.0.0.1:50714)
0620 20:16:38.148611 (+ 990us) negotiation.cc:107] Waiting for socket to connect
0620 20:16:38.148842 (+ 231us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.217.62:36319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":879,"thread_start_us":710,"threads_started":1}
W20260620 20:16:38.149479 24193 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.62:36319: connect: Connection refused (error 111)
I20260620 20:16:38.151093 24751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:38.150688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:36319 (local address 127.0.0.1:50724)
0620 20:16:38.150848 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0620 20:16:38.150927 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.217.62:36319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260620 20:16:38.151270 24193 leader_election.cc:334] T 00000000000000000000000000000000 P 5c48720af35c483da74e1243b72c87c3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 358753b177ca4441ba3b33b1e5fd7d66 (127.16.217.62:36319): Network error: Client connection negotiation failed: client connection to 127.16.217.62:36319: connect: Connection refused (error 111)
I20260620 20:16:38.155441 24753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:38.147796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59186 (local address 127.16.217.61:33441)
0620 20:16:38.148458 (+ 662us) server_negotiation.cc:206] Beginning negotiation
0620 20:16:38.148468 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 20:16:38.149043 (+ 575us) server_negotiation.cc:407] Connection header received
0620 20:16:38.149251 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:16:38.149257 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:16:38.149345 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:16:38.149430 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:16:38.150124 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:38.151244 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:38.153394 (+ 2150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:38.154468 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:38.154608 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:38.154710 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0620 20:16:38.154969 (+ 259us) server_negotiation.cc:299] Negotiation successful
0620 20:16:38.155097 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":487,"thread_start_us":252,"threads_started":1}
I20260620 20:16:38.155436 24752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:38.147714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.61:33441 (local address 127.0.0.1:59186)
0620 20:16:38.148927 (+ 1213us) negotiation.cc:107] Waiting for socket to connect
0620 20:16:38.148945 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 20:16:38.149094 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:16:38.149486 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:16:38.149493 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:16:38.149526 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:16:38.149918 (+ 392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:16:38.149936 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:16:38.151392 (+ 1456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:16:38.151399 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:16:38.153240 (+ 1841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:16:38.153257 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:16:38.154616 (+ 1359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:16:38.154622 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:16:38.154820 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:38.154844 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 20:16:38.154946 (+ 102us) client_negotiation.cc:240] Negotiation successful
0620 20:16:38.155079 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1091,"thread_start_us":930,"threads_started":1}
I20260620 20:16:38.156047 24025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5c48720af35c483da74e1243b72c87c3" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "0fe54b7d8d1b498f81774ec398b1f6aa" is_pre_election: true
I20260620 20:16:38.156422 24025 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 0fe54b7d8d1b498f81774ec398b1f6aa [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5c48720af35c483da74e1243b72c87c3 in term 1.
I20260620 20:16:38.156966 24191 leader_election.cc:302] T 00000000000000000000000000000000 P 5c48720af35c483da74e1243b72c87c3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0fe54b7d8d1b498f81774ec398b1f6aa, 5c48720af35c483da74e1243b72c87c3; no voters: 358753b177ca4441ba3b33b1e5fd7d66
I20260620 20:16:38.157752 24750 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5c48720af35c483da74e1243b72c87c3 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260620 20:16:38.157953 24750 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5c48720af35c483da74e1243b72c87c3 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 358753b177ca4441ba3b33b1e5fd7d66)
I20260620 20:16:38.158087 24750 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5c48720af35c483da74e1243b72c87c3 [term 1 FOLLOWER]: Advancing to term 2
I20260620 20:16:38.162199 24755 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0fe54b7d8d1b498f81774ec398b1f6aa [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 358753b177ca4441ba3b33b1e5fd7d66)
I20260620 20:16:38.162179 24750 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5c48720af35c483da74e1243b72c87c3 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "358753b177ca4441ba3b33b1e5fd7d66" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 36319 } } peers { permanent_uuid: "0fe54b7d8d1b498f81774ec398b1f6aa" member_type: VOTER last_known_addr { host: "127.16.217.61" port: 33441 } attrs { promote: false } } peers { permanent_uuid: "5c48720af35c483da74e1243b72c87c3" member_type: VOTER last_known_addr { host: "127.16.217.60" port: 37035 } attrs { promote: false } }
I20260620 20:16:38.162385 24755 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0fe54b7d8d1b498f81774ec398b1f6aa [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "358753b177ca4441ba3b33b1e5fd7d66" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 36319 } } peers { permanent_uuid: "0fe54b7d8d1b498f81774ec398b1f6aa" member_type: VOTER last_known_addr { host: "127.16.217.61" port: 33441 } attrs { promote: false } } peers { permanent_uuid: "5c48720af35c483da74e1243b72c87c3" member_type: VOTER last_known_addr { host: "127.16.217.60" port: 37035 } attrs { promote: false } }
I20260620 20:16:38.162894 24750 leader_election.cc:288] T 00000000000000000000000000000000 P 5c48720af35c483da74e1243b72c87c3 [CANDIDATE]: Term 2 election: Requested vote from peers 358753b177ca4441ba3b33b1e5fd7d66 (127.16.217.62:36319), 0fe54b7d8d1b498f81774ec398b1f6aa (127.16.217.61:33441)
I20260620 20:16:38.163439 24752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:38.163033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:36319 (local address 127.0.0.1:50744)
0620 20:16:38.163170 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0620 20:16:38.163275 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.217.62:36319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260620 20:16:38.163465 24025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5c48720af35c483da74e1243b72c87c3" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "0fe54b7d8d1b498f81774ec398b1f6aa"
I20260620 20:16:38.163662 24025 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0fe54b7d8d1b498f81774ec398b1f6aa [term 1 FOLLOWER]: Advancing to term 2
I20260620 20:16:38.164059 24755 leader_election.cc:288] T 00000000000000000000000000000000 P 0fe54b7d8d1b498f81774ec398b1f6aa [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 358753b177ca4441ba3b33b1e5fd7d66 (127.16.217.62:36319), 5c48720af35c483da74e1243b72c87c3 (127.16.217.60:37035)
I20260620 20:16:38.167858 24025 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 0fe54b7d8d1b498f81774ec398b1f6aa [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5c48720af35c483da74e1243b72c87c3 in term 2.
I20260620 20:16:38.168663 24191 leader_election.cc:302] T 00000000000000000000000000000000 P 5c48720af35c483da74e1243b72c87c3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fe54b7d8d1b498f81774ec398b1f6aa, 5c48720af35c483da74e1243b72c87c3; no voters:
I20260620 20:16:38.169071 24750 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5c48720af35c483da74e1243b72c87c3 [term 2 FOLLOWER]: Leader election won for term 2
I20260620 20:16:38.171092 24752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:38.170567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:36319 (local address 127.0.0.1:50762)
0620 20:16:38.170717 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0620 20:16:38.170849 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.217.62:36319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260620 20:16:38.171371 24193 leader_election.cc:334] T 00000000000000000000000000000000 P 5c48720af35c483da74e1243b72c87c3 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 358753b177ca4441ba3b33b1e5fd7d66 (127.16.217.62:36319): Network error: Client connection negotiation failed: client connection to 127.16.217.62:36319: connect: Connection refused (error 111)
I20260620 20:16:38.174736 24750 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5c48720af35c483da74e1243b72c87c3 [term 2 LEADER]: Becoming Leader. State: Replica: 5c48720af35c483da74e1243b72c87c3, State: Running, Role: LEADER
I20260620 20:16:38.175290 24750 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5c48720af35c483da74e1243b72c87c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "358753b177ca4441ba3b33b1e5fd7d66" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 36319 } } peers { permanent_uuid: "0fe54b7d8d1b498f81774ec398b1f6aa" member_type: VOTER last_known_addr { host: "127.16.217.61" port: 33441 } attrs { promote: false } } peers { permanent_uuid: "5c48720af35c483da74e1243b72c87c3" member_type: VOTER last_known_addr { host: "127.16.217.60" port: 37035 } attrs { promote: false } }
I20260620 20:16:38.177525 24756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:38.164280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:36319 (local address 127.0.0.1:50758)
0620 20:16:38.176920 (+ 12640us) negotiation.cc:107] Waiting for socket to connect
0620 20:16:38.177168 (+ 248us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.217.62:36319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":12453,"threads_started":1}
W20260620 20:16:38.178500 24002 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.62:36319: connect: Connection refused (error 111)
I20260620 20:16:38.183569 24758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:38.182915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:36319 (local address 127.0.0.1:50764)
0620 20:16:38.183098 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0620 20:16:38.183315 (+ 217us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.217.62:36319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260620 20:16:38.183744 24002 leader_election.cc:334] T 00000000000000000000000000000000 P 0fe54b7d8d1b498f81774ec398b1f6aa [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 358753b177ca4441ba3b33b1e5fd7d66 (127.16.217.62:36319): Network error: Client connection negotiation failed: client connection to 127.16.217.62:36319: connect: Connection refused (error 111)
I20260620 20:16:38.184247 24759 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c48720af35c483da74e1243b72c87c3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5c48720af35c483da74e1243b72c87c3. Latest consensus state: current_term: 2 leader_uuid: "5c48720af35c483da74e1243b72c87c3" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "358753b177ca4441ba3b33b1e5fd7d66" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 36319 } } peers { permanent_uuid: "0fe54b7d8d1b498f81774ec398b1f6aa" member_type: VOTER last_known_addr { host: "127.16.217.61" port: 33441 } attrs { promote: false } } peers { permanent_uuid: "5c48720af35c483da74e1243b72c87c3" member_type: VOTER last_known_addr { host: "127.16.217.60" port: 37035 } attrs { promote: false } } }
I20260620 20:16:38.184610 24759 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c48720af35c483da74e1243b72c87c3 [sys.catalog]: This master's current role is: LEADER
I20260620 20:16:38.185250 24762 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 20:16:38.187507 24756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:38.164488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.60:37035 (local address 127.0.0.1:49000)
0620 20:16:38.177746 (+ 13258us) negotiation.cc:107] Waiting for socket to connect
0620 20:16:38.177769 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 20:16:38.177954 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:16:38.180507 (+ 2553us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:16:38.180515 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:16:38.180554 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:16:38.180929 (+ 375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:16:38.180945 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:16:38.182883 (+ 1938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:16:38.182892 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:16:38.185230 (+ 2338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:16:38.185241 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:16:38.186969 (+ 1728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:16:38.186974 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:16:38.187101 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:38.187126 (+ 25us) client_negotiation.cc:769] Sending connection context
0620 20:16:38.187223 (+ 97us) client_negotiation.cc:240] Negotiation successful
0620 20:16:38.187315 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13149,"thread_start_us":14995,"threads_started":1}
I20260620 20:16:38.187822 24757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:38.164446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49000 (local address 127.16.217.60:37035)
0620 20:16:38.180096 (+ 15650us) server_negotiation.cc:206] Beginning negotiation
0620 20:16:38.180105 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 20:16:38.180134 (+ 29us) server_negotiation.cc:407] Connection header received
0620 20:16:38.180214 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:16:38.180222 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:16:38.180316 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:16:38.180449 (+ 133us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:16:38.181156 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:38.182596 (+ 1440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:38.185428 (+ 2832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:38.186782 (+ 1354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:38.186913 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:38.187022 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0620 20:16:38.187282 (+ 260us) server_negotiation.cc:299] Negotiation successful
0620 20:16:38.187413 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15483,"thread_start_us":16291,"threads_started":1}
I20260620 20:16:38.189570 24762 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 20:16:38.189652 24216 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0fe54b7d8d1b498f81774ec398b1f6aa" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "5c48720af35c483da74e1243b72c87c3" is_pre_election: true
I20260620 20:16:38.190404 24001 leader_election.cc:302] T 00000000000000000000000000000000 P 0fe54b7d8d1b498f81774ec398b1f6aa [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fe54b7d8d1b498f81774ec398b1f6aa; no voters: 358753b177ca4441ba3b33b1e5fd7d66, 5c48720af35c483da74e1243b72c87c3
I20260620 20:16:38.190476 24762 catalog_manager.cc:1259] Loaded cluster ID: a002c6e7a28743d2b82d56e2d2592ade
I20260620 20:16:38.190575 24762 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 20:16:38.190737 24755 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 0fe54b7d8d1b498f81774ec398b1f6aa [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 20:16:38.191977 24762 catalog_manager.cc:1504] Loading token signing keys...
I20260620 20:16:38.192760 24762 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 5c48720af35c483da74e1243b72c87c3: Loaded TSK: 0
I20260620 20:16:38.193606 24762 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 20:16:38.212625 24205 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48986:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 20:16:38.219638 24205 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 20:16:38.223976 24752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:38.223540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:36319 (local address 127.0.0.1:50778)
0620 20:16:38.223680 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0620 20:16:38.223800 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.217.62:36319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260620 20:16:38.224071 24025 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 0fe54b7d8d1b498f81774ec398b1f6aa [term 2 FOLLOWER]: Refusing update from remote peer 5c48720af35c483da74e1243b72c87c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260620 20:16:38.224850 24759 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5c48720af35c483da74e1243b72c87c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0fe54b7d8d1b498f81774ec398b1f6aa" member_type: VOTER last_known_addr { host: "127.16.217.61" port: 33441 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260620 20:16:38.224989 24752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:38.224592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:36319 (local address 127.0.0.1:50792)
0620 20:16:38.224786 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0620 20:16:38.224863 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.217.62:36319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260620 20:16:38.225203 24193 consensus_peers.cc:487] T 00000000000000000000000000000000 P 5c48720af35c483da74e1243b72c87c3 -> Peer 358753b177ca4441ba3b33b1e5fd7d66 (127.16.217.62:36319): Couldn't send request to peer 358753b177ca4441ba3b33b1e5fd7d66. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.62:36319: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260620 20:16:38.228582 24755 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0fe54b7d8d1b498f81774ec398b1f6aa [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5c48720af35c483da74e1243b72c87c3. Latest consensus state: current_term: 2 leader_uuid: "5c48720af35c483da74e1243b72c87c3" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "358753b177ca4441ba3b33b1e5fd7d66" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 36319 } } peers { permanent_uuid: "0fe54b7d8d1b498f81774ec398b1f6aa" member_type: VOTER last_known_addr { host: "127.16.217.61" port: 33441 } attrs { promote: false } } peers { permanent_uuid: "5c48720af35c483da74e1243b72c87c3" member_type: VOTER last_known_addr { host: "127.16.217.60" port: 37035 } attrs { promote: false } } }
I20260620 20:16:38.228922 24755 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0fe54b7d8d1b498f81774ec398b1f6aa [sys.catalog]: This master's current role is: FOLLOWER
I20260620 20:16:38.229980 24759 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c48720af35c483da74e1243b72c87c3 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "5c48720af35c483da74e1243b72c87c3" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "358753b177ca4441ba3b33b1e5fd7d66" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 36319 } } peers { permanent_uuid: "0fe54b7d8d1b498f81774ec398b1f6aa" member_type: VOTER last_known_addr { host: "127.16.217.61" port: 33441 } attrs { promote: false } } peers { permanent_uuid: "5c48720af35c483da74e1243b72c87c3" member_type: VOTER last_known_addr { host: "127.16.217.60" port: 37035 } attrs { promote: false } } }
I20260620 20:16:38.230296 24759 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c48720af35c483da74e1243b72c87c3 [sys.catalog]: This master's current role is: LEADER
I20260620 20:16:38.230764 24755 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0fe54b7d8d1b498f81774ec398b1f6aa [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "5c48720af35c483da74e1243b72c87c3" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "358753b177ca4441ba3b33b1e5fd7d66" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 36319 } } peers { permanent_uuid: "0fe54b7d8d1b498f81774ec398b1f6aa" member_type: VOTER last_known_addr { host: "127.16.217.61" port: 33441 } attrs { promote: false } } peers { permanent_uuid: "5c48720af35c483da74e1243b72c87c3" member_type: VOTER last_known_addr { host: "127.16.217.60" port: 37035 } attrs { promote: false } } }
I20260620 20:16:38.231104 24755 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0fe54b7d8d1b498f81774ec398b1f6aa [sys.catalog]: This master's current role is: FOLLOWER
I20260620 20:16:38.245327 24752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:38.238984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.1:34627 (local address 127.0.0.1:35964)
0620 20:16:38.239229 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0620 20:16:38.239252 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 20:16:38.239354 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:16:38.239911 (+ 557us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:16:38.239916 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:16:38.239942 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:16:38.240129 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:16:38.240139 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:16:38.241585 (+ 1446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:16:38.241591 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:16:38.243095 (+ 1504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:16:38.243106 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:16:38.244480 (+ 1374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:16:38.244485 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:16:38.244702 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:38.244749 (+ 47us) client_negotiation.cc:769] Sending connection context
0620 20:16:38.244839 (+ 90us) client_negotiation.cc:240] Negotiation successful
0620 20:16:38.245072 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"mutex_wait_us":98}
I20260620 20:16:38.246317 24768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:38.239125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35964 (local address 127.16.217.1:34627)
0620 20:16:38.239589 (+ 464us) server_negotiation.cc:206] Beginning negotiation
0620 20:16:38.239594 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:16:38.239609 (+ 15us) server_negotiation.cc:407] Connection header received
0620 20:16:38.239666 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:16:38.239669 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:16:38.239762 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:16:38.239881 (+ 119us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:16:38.240313 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:38.241378 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:38.243273 (+ 1895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:38.244320 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:38.245751 (+ 1431us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:38.245844 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0620 20:16:38.245942 (+ 98us) server_negotiation.cc:299] Negotiation successful
0620 20:16:38.246042 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":365,"thread_start_us":279,"threads_started":1}
I20260620 20:16:38.251176 24751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:38.239374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.3:46165 (local address 127.0.0.1:53924)
0620 20:16:38.239759 (+ 385us) negotiation.cc:107] Waiting for socket to connect
0620 20:16:38.239777 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 20:16:38.239885 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:16:38.241577 (+ 1692us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:16:38.241586 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:16:38.241622 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:16:38.242135 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:16:38.242156 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:16:38.244256 (+ 2100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:16:38.244265 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:16:38.247474 (+ 3209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:16:38.247496 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:16:38.250038 (+ 2542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:16:38.250049 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:16:38.250354 (+ 305us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:38.250417 (+ 63us) client_negotiation.cc:769] Sending connection context
0620 20:16:38.250509 (+ 92us) client_negotiation.cc:240] Negotiation successful
0620 20:16:38.250731 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284}
I20260620 20:16:38.251950 24752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:38.240979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.2:41479 (local address 127.0.0.1:48686)
0620 20:16:38.245554 (+ 4575us) negotiation.cc:107] Waiting for socket to connect
0620 20:16:38.245571 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 20:16:38.245674 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:16:38.246797 (+ 1123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:16:38.246801 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:16:38.246822 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:16:38.247035 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:16:38.247044 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:16:38.248335 (+ 1291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:16:38.248343 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:16:38.249864 (+ 1521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:16:38.249882 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:16:38.251167 (+ 1285us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:16:38.251174 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:16:38.251424 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:38.251455 (+ 31us) client_negotiation.cc:769] Sending connection context
0620 20:16:38.251547 (+ 92us) client_negotiation.cc:240] Negotiation successful
0620 20:16:38.251708 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4492,"threads_started":1}
I20260620 20:16:38.252424 24769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:38.239757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53924 (local address 127.16.217.3:46165)
0620 20:16:38.241165 (+ 1408us) server_negotiation.cc:206] Beginning negotiation
0620 20:16:38.241170 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:16:38.241191 (+ 21us) server_negotiation.cc:407] Connection header received
0620 20:16:38.241282 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:16:38.241287 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:16:38.241373 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:16:38.242370 (+ 997us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:16:38.242600 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:38.244032 (+ 1432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:38.247807 (+ 3775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:38.249653 (+ 1846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:38.251693 (+ 2040us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:38.251831 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0620 20:16:38.251890 (+ 59us) server_negotiation.cc:299] Negotiation successful
0620 20:16:38.252049 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1267,"thread_start_us":1941,"threads_started":1}
I20260620 20:16:38.252516 24771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:38.241345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48686 (local address 127.16.217.2:41479)
0620 20:16:38.241835 (+ 490us) server_negotiation.cc:206] Beginning negotiation
0620 20:16:38.241842 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:16:38.246511 (+ 4669us) server_negotiation.cc:407] Connection header received
0620 20:16:38.246589 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:16:38.246593 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:16:38.246669 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:16:38.246783 (+ 114us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:16:38.247335 (+ 552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:38.248166 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:38.250166 (+ 2000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:38.250900 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:38.251095 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:38.251216 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0620 20:16:38.252155 (+ 939us) server_negotiation.cc:299] Negotiation successful
0620 20:16:38.252288 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":365,"thread_start_us":258,"threads_started":1}
I20260620 20:16:38.252707 24321 tablet_service.cc:1467] Processing CreateTablet for tablet eda5a33f39884e50bf209c048ff3b717 (DEFAULT_TABLE table=test-table [id=6134826cd44a419cb9d5fb47a9d6b4fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:16:38.253261 24321 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eda5a33f39884e50bf209c048ff3b717. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:16:38.253228 24610 tablet_service.cc:1467] Processing CreateTablet for tablet eda5a33f39884e50bf209c048ff3b717 (DEFAULT_TABLE table=test-table [id=6134826cd44a419cb9d5fb47a9d6b4fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:16:38.253654 24610 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eda5a33f39884e50bf209c048ff3b717. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:16:38.258731 24772 tablet_bootstrap.cc:492] T eda5a33f39884e50bf209c048ff3b717 P b6d268e1e2e944af9e30e4b3e4b3e359: Bootstrap starting.
I20260620 20:16:38.258872 24468 tablet_service.cc:1467] Processing CreateTablet for tablet eda5a33f39884e50bf209c048ff3b717 (DEFAULT_TABLE table=test-table [id=6134826cd44a419cb9d5fb47a9d6b4fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:16:38.259454 24468 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eda5a33f39884e50bf209c048ff3b717. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:16:38.259593 24773 tablet_bootstrap.cc:492] T eda5a33f39884e50bf209c048ff3b717 P 5280c2f4f3ef4c77890e5b5b7c7f78ae: Bootstrap starting.
I20260620 20:16:38.260597 24772 tablet_bootstrap.cc:654] T eda5a33f39884e50bf209c048ff3b717 P b6d268e1e2e944af9e30e4b3e4b3e359: Neither blocks nor log segments found. Creating new log.
I20260620 20:16:38.261335 24773 tablet_bootstrap.cc:654] T eda5a33f39884e50bf209c048ff3b717 P 5280c2f4f3ef4c77890e5b5b7c7f78ae: Neither blocks nor log segments found. Creating new log.
I20260620 20:16:38.263019 24772 tablet_bootstrap.cc:492] T eda5a33f39884e50bf209c048ff3b717 P b6d268e1e2e944af9e30e4b3e4b3e359: No bootstrap required, opened a new log
I20260620 20:16:38.263175 24772 ts_tablet_manager.cc:1397] T eda5a33f39884e50bf209c048ff3b717 P b6d268e1e2e944af9e30e4b3e4b3e359: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260620 20:16:38.264030 24772 raft_consensus.cc:348] T eda5a33f39884e50bf209c048ff3b717 P b6d268e1e2e944af9e30e4b3e4b3e359 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5280c2f4f3ef4c77890e5b5b7c7f78ae" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 34627 } } peers { permanent_uuid: "b6d268e1e2e944af9e30e4b3e4b3e359" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 46165 } } peers { permanent_uuid: "33fb52c1b00347a8ac5db344e2959cac" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 41479 } }
I20260620 20:16:38.264297 24772 raft_consensus.cc:374] T eda5a33f39884e50bf209c048ff3b717 P b6d268e1e2e944af9e30e4b3e4b3e359 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:16:38.264405 24772 raft_consensus.cc:729] T eda5a33f39884e50bf209c048ff3b717 P b6d268e1e2e944af9e30e4b3e4b3e359 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6d268e1e2e944af9e30e4b3e4b3e359, State: Initialized, Role: FOLLOWER
I20260620 20:16:38.264621 24773 tablet_bootstrap.cc:492] T eda5a33f39884e50bf209c048ff3b717 P 5280c2f4f3ef4c77890e5b5b7c7f78ae: No bootstrap required, opened a new log
I20260620 20:16:38.264791 24773 ts_tablet_manager.cc:1397] T eda5a33f39884e50bf209c048ff3b717 P 5280c2f4f3ef4c77890e5b5b7c7f78ae: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20260620 20:16:38.264636 24772 consensus_queue.cc:260] T eda5a33f39884e50bf209c048ff3b717 P b6d268e1e2e944af9e30e4b3e4b3e359 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5280c2f4f3ef4c77890e5b5b7c7f78ae" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 34627 } } peers { permanent_uuid: "b6d268e1e2e944af9e30e4b3e4b3e359" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 46165 } } peers { permanent_uuid: "33fb52c1b00347a8ac5db344e2959cac" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 41479 } }
I20260620 20:16:38.265492 24772 ts_tablet_manager.cc:1428] T eda5a33f39884e50bf209c048ff3b717 P b6d268e1e2e944af9e30e4b3e4b3e359: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260620 20:16:38.265714 24773 raft_consensus.cc:348] T eda5a33f39884e50bf209c048ff3b717 P 5280c2f4f3ef4c77890e5b5b7c7f78ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5280c2f4f3ef4c77890e5b5b7c7f78ae" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 34627 } } peers { permanent_uuid: "b6d268e1e2e944af9e30e4b3e4b3e359" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 46165 } } peers { permanent_uuid: "33fb52c1b00347a8ac5db344e2959cac" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 41479 } }
I20260620 20:16:38.265978 24773 raft_consensus.cc:374] T eda5a33f39884e50bf209c048ff3b717 P 5280c2f4f3ef4c77890e5b5b7c7f78ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:16:38.266075 24773 raft_consensus.cc:729] T eda5a33f39884e50bf209c048ff3b717 P 5280c2f4f3ef4c77890e5b5b7c7f78ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5280c2f4f3ef4c77890e5b5b7c7f78ae, State: Initialized, Role: FOLLOWER
I20260620 20:16:38.266348 24773 consensus_queue.cc:260] T eda5a33f39884e50bf209c048ff3b717 P 5280c2f4f3ef4c77890e5b5b7c7f78ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5280c2f4f3ef4c77890e5b5b7c7f78ae" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 34627 } } peers { permanent_uuid: "b6d268e1e2e944af9e30e4b3e4b3e359" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 46165 } } peers { permanent_uuid: "33fb52c1b00347a8ac5db344e2959cac" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 41479 } }
I20260620 20:16:38.269774 24675 heartbeater.cc:503] Master 127.16.217.60:37035 was elected leader, sending a full tablet report...
I20260620 20:16:38.273284 24773 ts_tablet_manager.cc:1428] T eda5a33f39884e50bf209c048ff3b717 P 5280c2f4f3ef4c77890e5b5b7c7f78ae: Time spent starting tablet: real 0.008s user 0.004s sys 0.005s
I20260620 20:16:38.278204 24386 heartbeater.cc:503] Master 127.16.217.60:37035 was elected leader, sending a full tablet report...
I20260620 20:16:38.279722 24779 tablet_bootstrap.cc:492] T eda5a33f39884e50bf209c048ff3b717 P 33fb52c1b00347a8ac5db344e2959cac: Bootstrap starting.
I20260620 20:16:38.282279 24779 tablet_bootstrap.cc:654] T eda5a33f39884e50bf209c048ff3b717 P 33fb52c1b00347a8ac5db344e2959cac: Neither blocks nor log segments found. Creating new log.
I20260620 20:16:38.282444 24777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:38.266316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:36319 (local address 127.16.217.3:43719)
0620 20:16:38.281727 (+ 15411us) negotiation.cc:107] Waiting for socket to connect
0620 20:16:38.282110 (+ 383us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.217.62:36319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":15261,"thread_start_us":15577,"threads_started":1}
I20260620 20:16:38.284080 24777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:38.283133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:36319 (local address 127.16.217.3:37469)
0620 20:16:38.283843 (+ 710us) negotiation.cc:107] Waiting for socket to connect
0620 20:16:38.283932 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.217.62:36319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":616,"threads_started":1}
I20260620 20:16:38.284490 24780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:38.274122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:36319 (local address 127.16.217.1:58247)
0620 20:16:38.283946 (+ 9824us) negotiation.cc:107] Waiting for socket to connect
0620 20:16:38.284138 (+ 192us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.217.62:36319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":9646,"thread_start_us":9500,"threads_started":1}
I20260620 20:16:38.285810 24780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:38.285374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:36319 (local address 127.16.217.1:46189)
0620 20:16:38.285588 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0620 20:16:38.285673 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.217.62:36319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114}
I20260620 20:16:38.286732 24779 tablet_bootstrap.cc:492] T eda5a33f39884e50bf209c048ff3b717 P 33fb52c1b00347a8ac5db344e2959cac: No bootstrap required, opened a new log
I20260620 20:16:38.286893 24779 ts_tablet_manager.cc:1397] T eda5a33f39884e50bf209c048ff3b717 P 33fb52c1b00347a8ac5db344e2959cac: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260620 20:16:38.287511 24779 raft_consensus.cc:348] T eda5a33f39884e50bf209c048ff3b717 P 33fb52c1b00347a8ac5db344e2959cac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5280c2f4f3ef4c77890e5b5b7c7f78ae" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 34627 } } peers { permanent_uuid: "b6d268e1e2e944af9e30e4b3e4b3e359" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 46165 } } peers { permanent_uuid: "33fb52c1b00347a8ac5db344e2959cac" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 41479 } }
I20260620 20:16:38.287737 24779 raft_consensus.cc:374] T eda5a33f39884e50bf209c048ff3b717 P 33fb52c1b00347a8ac5db344e2959cac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:16:38.287853 24779 raft_consensus.cc:729] T eda5a33f39884e50bf209c048ff3b717 P 33fb52c1b00347a8ac5db344e2959cac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33fb52c1b00347a8ac5db344e2959cac, State: Initialized, Role: FOLLOWER
I20260620 20:16:38.288094 24779 consensus_queue.cc:260] T eda5a33f39884e50bf209c048ff3b717 P 33fb52c1b00347a8ac5db344e2959cac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5280c2f4f3ef4c77890e5b5b7c7f78ae" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 34627 } } peers { permanent_uuid: "b6d268e1e2e944af9e30e4b3e4b3e359" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 46165 } } peers { permanent_uuid: "33fb52c1b00347a8ac5db344e2959cac" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 41479 } }
I20260620 20:16:38.288831 24779 ts_tablet_manager.cc:1428] T eda5a33f39884e50bf209c048ff3b717 P 33fb52c1b00347a8ac5db344e2959cac: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 20:16:38.290135 24784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:38.289134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:36319 (local address 127.16.217.2:50011)
0620 20:16:38.289731 (+ 597us) negotiation.cc:107] Waiting for socket to connect
0620 20:16:38.289901 (+ 170us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.217.62:36319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":481,"thread_start_us":327,"threads_started":1}
I20260620 20:16:38.291319 24533 heartbeater.cc:503] Master 127.16.217.60:37035 was elected leader, sending a full tablet report...
I20260620 20:16:38.291846 24784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:38.290710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:36319 (local address 127.16.217.2:35369)
0620 20:16:38.291177 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0620 20:16:38.291281 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.217.62:36319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":759,"threads_started":1}
I20260620 20:16:38.345674 24778 raft_consensus.cc:482] T eda5a33f39884e50bf209c048ff3b717 P 5280c2f4f3ef4c77890e5b5b7c7f78ae [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 20:16:38.345978 24778 raft_consensus.cc:504] T eda5a33f39884e50bf209c048ff3b717 P 5280c2f4f3ef4c77890e5b5b7c7f78ae [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5280c2f4f3ef4c77890e5b5b7c7f78ae" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 34627 } } peers { permanent_uuid: "b6d268e1e2e944af9e30e4b3e4b3e359" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 46165 } } peers { permanent_uuid: "33fb52c1b00347a8ac5db344e2959cac" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 41479 } }
I20260620 20:16:38.347576 24778 leader_election.cc:288] T eda5a33f39884e50bf209c048ff3b717 P 5280c2f4f3ef4c77890e5b5b7c7f78ae [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b6d268e1e2e944af9e30e4b3e4b3e359 (127.16.217.3:46165), 33fb52c1b00347a8ac5db344e2959cac (127.16.217.2:41479)
I20260620 20:16:38.354159 24780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:38.347770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.3:46165 (local address 127.16.217.1:39263)
0620 20:16:38.348011 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0620 20:16:38.348033 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 20:16:38.348174 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:16:38.348599 (+ 425us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:16:38.348605 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:16:38.348632 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:16:38.348971 (+ 339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:16:38.348981 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:16:38.350091 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:16:38.350097 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:16:38.351742 (+ 1645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:16:38.351804 (+ 62us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:16:38.353307 (+ 1503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:16:38.353313 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:16:38.353494 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:38.353528 (+ 34us) client_negotiation.cc:769] Sending connection context
0620 20:16:38.353649 (+ 121us) client_negotiation.cc:240] Negotiation successful
0620 20:16:38.353801 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":98,"mutex_wait_us":2}
I20260620 20:16:38.354158 24769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:38.347980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.217.1:39263 (local address 127.16.217.3:46165)
0620 20:16:38.348213 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0620 20:16:38.348223 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 20:16:38.348252 (+ 29us) server_negotiation.cc:407] Connection header received
0620 20:16:38.348347 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:16:38.348356 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:16:38.348449 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:16:38.348558 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:16:38.349127 (+ 569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:38.349971 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:38.352008 (+ 2037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:38.353138 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:38.353282 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:38.353364 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0620 20:16:38.353682 (+ 318us) server_negotiation.cc:299] Negotiation successful
0620 20:16:38.353832 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 20:16:38.354938 24771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:38.348170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.217.1:35393 (local address 127.16.217.2:41479)
0620 20:16:38.348412 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0620 20:16:38.348420 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 20:16:38.348806 (+ 386us) server_negotiation.cc:407] Connection header received
0620 20:16:38.348952 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:16:38.348956 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:16:38.349013 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:16:38.349144 (+ 131us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:16:38.349629 (+ 485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:38.350582 (+ 953us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:38.352827 (+ 2245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:38.353896 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:38.354042 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:38.354104 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0620 20:16:38.354601 (+ 497us) server_negotiation.cc:299] Negotiation successful
0620 20:16:38.354738 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260620 20:16:38.355058 24630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eda5a33f39884e50bf209c048ff3b717" candidate_uuid: "5280c2f4f3ef4c77890e5b5b7c7f78ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b6d268e1e2e944af9e30e4b3e4b3e359" is_pre_election: true
I20260620 20:16:38.355523 24786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:38.347975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.2:41479 (local address 127.16.217.1:35393)
0620 20:16:38.348673 (+ 698us) negotiation.cc:107] Waiting for socket to connect
0620 20:16:38.348692 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 20:16:38.348825 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:16:38.349160 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:16:38.349163 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:16:38.349181 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:16:38.349458 (+ 277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:16:38.349465 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:16:38.350738 (+ 1273us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:16:38.350747 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:16:38.352660 (+ 1913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:16:38.352675 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:16:38.354104 (+ 1429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:16:38.354110 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:16:38.354405 (+ 295us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:38.354446 (+ 41us) client_negotiation.cc:769] Sending connection context
0620 20:16:38.355110 (+ 664us) client_negotiation.cc:240] Negotiation successful
0620 20:16:38.355206 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":586,"thread_start_us":415,"threads_started":1}
I20260620 20:16:38.355458 24630 raft_consensus.cc:2454] T eda5a33f39884e50bf209c048ff3b717 P b6d268e1e2e944af9e30e4b3e4b3e359 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5280c2f4f3ef4c77890e5b5b7c7f78ae in term 0.
I20260620 20:16:38.356062 24488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eda5a33f39884e50bf209c048ff3b717" candidate_uuid: "5280c2f4f3ef4c77890e5b5b7c7f78ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33fb52c1b00347a8ac5db344e2959cac" is_pre_election: true
I20260620 20:16:38.356067 24276 leader_election.cc:302] T eda5a33f39884e50bf209c048ff3b717 P 5280c2f4f3ef4c77890e5b5b7c7f78ae [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5280c2f4f3ef4c77890e5b5b7c7f78ae, b6d268e1e2e944af9e30e4b3e4b3e359; no voters:
I20260620 20:16:38.356273 24488 raft_consensus.cc:2454] T eda5a33f39884e50bf209c048ff3b717 P 33fb52c1b00347a8ac5db344e2959cac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5280c2f4f3ef4c77890e5b5b7c7f78ae in term 0.
I20260620 20:16:38.356431 24778 raft_consensus.cc:2781] T eda5a33f39884e50bf209c048ff3b717 P 5280c2f4f3ef4c77890e5b5b7c7f78ae [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 20:16:38.356571 24778 raft_consensus.cc:482] T eda5a33f39884e50bf209c048ff3b717 P 5280c2f4f3ef4c77890e5b5b7c7f78ae [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 20:16:38.356700 24778 raft_consensus.cc:3037] T eda5a33f39884e50bf209c048ff3b717 P 5280c2f4f3ef4c77890e5b5b7c7f78ae [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:16:38.358806 24778 raft_consensus.cc:504] T eda5a33f39884e50bf209c048ff3b717 P 5280c2f4f3ef4c77890e5b5b7c7f78ae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5280c2f4f3ef4c77890e5b5b7c7f78ae" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 34627 } } peers { permanent_uuid: "b6d268e1e2e944af9e30e4b3e4b3e359" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 46165 } } peers { permanent_uuid: "33fb52c1b00347a8ac5db344e2959cac" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 41479 } }
I20260620 20:16:38.359406 24778 leader_election.cc:288] T eda5a33f39884e50bf209c048ff3b717 P 5280c2f4f3ef4c77890e5b5b7c7f78ae [CANDIDATE]: Term 1 election: Requested vote from peers b6d268e1e2e944af9e30e4b3e4b3e359 (127.16.217.3:46165), 33fb52c1b00347a8ac5db344e2959cac (127.16.217.2:41479)
I20260620 20:16:38.359804 24630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eda5a33f39884e50bf209c048ff3b717" candidate_uuid: "5280c2f4f3ef4c77890e5b5b7c7f78ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b6d268e1e2e944af9e30e4b3e4b3e359"
I20260620 20:16:38.359930 24488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eda5a33f39884e50bf209c048ff3b717" candidate_uuid: "5280c2f4f3ef4c77890e5b5b7c7f78ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33fb52c1b00347a8ac5db344e2959cac"
I20260620 20:16:38.360041 24630 raft_consensus.cc:3037] T eda5a33f39884e50bf209c048ff3b717 P b6d268e1e2e944af9e30e4b3e4b3e359 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:16:38.360122 24488 raft_consensus.cc:3037] T eda5a33f39884e50bf209c048ff3b717 P 33fb52c1b00347a8ac5db344e2959cac [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:16:38.363031 24488 raft_consensus.cc:2454] T eda5a33f39884e50bf209c048ff3b717 P 33fb52c1b00347a8ac5db344e2959cac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5280c2f4f3ef4c77890e5b5b7c7f78ae in term 1.
I20260620 20:16:38.363032 24630 raft_consensus.cc:2454] T eda5a33f39884e50bf209c048ff3b717 P b6d268e1e2e944af9e30e4b3e4b3e359 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5280c2f4f3ef4c77890e5b5b7c7f78ae in term 1.
I20260620 20:16:38.363394 24276 leader_election.cc:302] T eda5a33f39884e50bf209c048ff3b717 P 5280c2f4f3ef4c77890e5b5b7c7f78ae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5280c2f4f3ef4c77890e5b5b7c7f78ae, b6d268e1e2e944af9e30e4b3e4b3e359; no voters:
I20260620 20:16:38.363646 24778 raft_consensus.cc:2781] T eda5a33f39884e50bf209c048ff3b717 P 5280c2f4f3ef4c77890e5b5b7c7f78ae [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:16:38.364188 24778 raft_consensus.cc:686] T eda5a33f39884e50bf209c048ff3b717 P 5280c2f4f3ef4c77890e5b5b7c7f78ae [term 1 LEADER]: Becoming Leader. State: Replica: 5280c2f4f3ef4c77890e5b5b7c7f78ae, State: Running, Role: LEADER
I20260620 20:16:38.364596 24778 consensus_queue.cc:237] T eda5a33f39884e50bf209c048ff3b717 P 5280c2f4f3ef4c77890e5b5b7c7f78ae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5280c2f4f3ef4c77890e5b5b7c7f78ae" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 34627 } } peers { permanent_uuid: "b6d268e1e2e944af9e30e4b3e4b3e359" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 46165 } } peers { permanent_uuid: "33fb52c1b00347a8ac5db344e2959cac" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 41479 } }
I20260620 20:16:38.365162 24786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:38.364732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:36319 (local address 127.16.217.1:58245)
0620 20:16:38.364947 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0620 20:16:38.365037 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.217.62:36319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260620 20:16:38.368016 24205 catalog_manager.cc:5495] T eda5a33f39884e50bf209c048ff3b717 P 5280c2f4f3ef4c77890e5b5b7c7f78ae reported cstate change: term changed from 0 to 1, leader changed from <none> to 5280c2f4f3ef4c77890e5b5b7c7f78ae (127.16.217.1). New cstate: current_term: 1 leader_uuid: "5280c2f4f3ef4c77890e5b5b7c7f78ae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5280c2f4f3ef4c77890e5b5b7c7f78ae" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 34627 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b6d268e1e2e944af9e30e4b3e4b3e359" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 46165 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "33fb52c1b00347a8ac5db344e2959cac" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 41479 } health_report { overall_health: UNKNOWN } } }
I20260620 20:16:38.405298 24768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:38.400194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35972 (local address 127.16.217.1:34627)
0620 20:16:38.400396 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0620 20:16:38.400402 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:16:38.400421 (+ 19us) server_negotiation.cc:407] Connection header received
0620 20:16:38.400619 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:16:38.400623 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:16:38.400711 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:16:38.400873 (+ 162us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 20:16:38.401225 (+ 352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:38.401988 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:38.403276 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:38.403693 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:38.403825 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:38.404223 (+ 398us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 20:16:38.404749 (+ 526us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 20:16:38.404906 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0620 20:16:38.404960 (+ 54us) server_negotiation.cc:299] Negotiation successful
0620 20:16:38.405038 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 20:16:38.409456 24488 raft_consensus.cc:1261] T eda5a33f39884e50bf209c048ff3b717 P 33fb52c1b00347a8ac5db344e2959cac [term 1 FOLLOWER]: Refusing update from remote peer 5280c2f4f3ef4c77890e5b5b7c7f78ae: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:16:38.409984 24630 raft_consensus.cc:1261] T eda5a33f39884e50bf209c048ff3b717 P b6d268e1e2e944af9e30e4b3e4b3e359 [term 1 FOLLOWER]: Refusing update from remote peer 5280c2f4f3ef4c77890e5b5b7c7f78ae: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:16:38.410207 24787 consensus_queue.cc:1035] T eda5a33f39884e50bf209c048ff3b717 P 5280c2f4f3ef4c77890e5b5b7c7f78ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "33fb52c1b00347a8ac5db344e2959cac" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 41479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:16:38.410704 24778 consensus_queue.cc:1035] T eda5a33f39884e50bf209c048ff3b717 P 5280c2f4f3ef4c77890e5b5b7c7f78ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6d268e1e2e944af9e30e4b3e4b3e359" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 46165 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:16:38.412971 24784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:38.412429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:36319 (local address 127.16.217.2:52425)
0620 20:16:38.412702 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0620 20:16:38.412786 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.217.62:36319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143}
I20260620 20:16:38.414227 24782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:38.413439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:36319 (local address 127.16.217.3:51361)
0620 20:16:38.413710 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0620 20:16:38.413990 (+ 280us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.217.62:36319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139}
I20260620 20:16:38.423851 24782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:38.423365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:36319 (local address 127.16.217.3:58207)
0620 20:16:38.423607 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0620 20:16:38.423704 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.217.62:36319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144}
I20260620 20:16:38.425992 24786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:38.425309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:36319 (local address 127.16.217.1:43555)
0620 20:16:38.425623 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0620 20:16:38.425746 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.217.62:36319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178}
I20260620 20:16:38.426779 24784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:38.425120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:36319 (local address 127.16.217.2:47119)
0620 20:16:38.426457 (+ 1337us) negotiation.cc:107] Waiting for socket to connect
0620 20:16:38.426570 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.217.62:36319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1194}
I20260620 20:16:38.429718 24792 mvcc.cc:205] Tried to move back new op lower bound from 7299017107078615040 to 7299017106905227264. Current Snapshot: MvccSnapshot[applied={T|T < 7299017107078615040}]
I20260620 20:16:38.439482 24206 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48986:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 20:16:38.439785 24206 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48986:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 20:16:38.445670 24206 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5c48720af35c483da74e1243b72c87c3: Sending DeleteTablet for 3 replicas of tablet eda5a33f39884e50bf209c048ff3b717
I20260620 20:16:38.446599 24321 tablet_service.cc:1514] Processing DeleteTablet for tablet eda5a33f39884e50bf209c048ff3b717 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:16:38 UTC) from {username='oryx', principal='oryx/127.16.217.60@KRBTEST.COM'} at 127.0.0.1:35964
I20260620 20:16:38.446964 24610 tablet_service.cc:1514] Processing DeleteTablet for tablet eda5a33f39884e50bf209c048ff3b717 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:16:38 UTC) from {username='oryx', principal='oryx/127.16.217.60@KRBTEST.COM'} at 127.0.0.1:53924
I20260620 20:16:38.447412 24468 tablet_service.cc:1514] Processing DeleteTablet for tablet eda5a33f39884e50bf209c048ff3b717 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:16:38 UTC) from {username='oryx', principal='oryx/127.16.217.60@KRBTEST.COM'} at 127.0.0.1:48686
I20260620 20:16:38.448127 24802 tablet_replica.cc:330] T eda5a33f39884e50bf209c048ff3b717 P 33fb52c1b00347a8ac5db344e2959cac: stopping tablet replica
I20260620 20:16:38.448379 24802 raft_consensus.cc:2229] T eda5a33f39884e50bf209c048ff3b717 P 33fb52c1b00347a8ac5db344e2959cac [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 20:16:38.448583 17252 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu pid 24243
I20260620 20:16:38.448707 24802 raft_consensus.cc:2258] T eda5a33f39884e50bf209c048ff3b717 P 33fb52c1b00347a8ac5db344e2959cac [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:16:38.456115 24802 ts_tablet_manager.cc:1905] T eda5a33f39884e50bf209c048ff3b717 P 33fb52c1b00347a8ac5db344e2959cac: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:16:38.458539 24800 tablet_replica.cc:330] T eda5a33f39884e50bf209c048ff3b717 P 5280c2f4f3ef4c77890e5b5b7c7f78ae: stopping tablet replica
I20260620 20:16:38.458861 24800 raft_consensus.cc:2229] T eda5a33f39884e50bf209c048ff3b717 P 5280c2f4f3ef4c77890e5b5b7c7f78ae [term 1 LEADER]: Raft consensus shutting down.
I20260620 20:16:38.459530 24800 raft_consensus.cc:2258] T eda5a33f39884e50bf209c048ff3b717 P 5280c2f4f3ef4c77890e5b5b7c7f78ae [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:16:38.461010 24800 ts_tablet_manager.cc:1905] T eda5a33f39884e50bf209c048ff3b717 P 5280c2f4f3ef4c77890e5b5b7c7f78ae: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:16:38.461959 24801 tablet_replica.cc:330] T eda5a33f39884e50bf209c048ff3b717 P b6d268e1e2e944af9e30e4b3e4b3e359: stopping tablet replica
I20260620 20:16:38.462169 24801 raft_consensus.cc:2229] T eda5a33f39884e50bf209c048ff3b717 P b6d268e1e2e944af9e30e4b3e4b3e359 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 20:16:38.462463 24801 raft_consensus.cc:2258] T eda5a33f39884e50bf209c048ff3b717 P b6d268e1e2e944af9e30e4b3e4b3e359 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:16:38.463649 24801 ts_tablet_manager.cc:1905] T eda5a33f39884e50bf209c048ff3b717 P b6d268e1e2e944af9e30e4b3e4b3e359: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:16:38.468820 24800 ts_tablet_manager.cc:1918] T eda5a33f39884e50bf209c048ff3b717 P 5280c2f4f3ef4c77890e5b5b7c7f78ae: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 20:16:38.468962 24800 log.cc:1196] T eda5a33f39884e50bf209c048ff3b717 P 5280c2f4f3ef4c77890e5b5b7c7f78ae: Deleting WAL directory at /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986536687152-17252-0/minicluster-data/ts-0/wal/wals/eda5a33f39884e50bf209c048ff3b717
I20260620 20:16:38.469448 24800 ts_tablet_manager.cc:1939] T eda5a33f39884e50bf209c048ff3b717 P 5280c2f4f3ef4c77890e5b5b7c7f78ae: Deleting consensus metadata
I20260620 20:16:38.469655 24802 ts_tablet_manager.cc:1918] T eda5a33f39884e50bf209c048ff3b717 P 33fb52c1b00347a8ac5db344e2959cac: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 20:16:38.469796 24802 log.cc:1196] T eda5a33f39884e50bf209c048ff3b717 P 33fb52c1b00347a8ac5db344e2959cac: Deleting WAL directory at /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986536687152-17252-0/minicluster-data/ts-1/wal/wals/eda5a33f39884e50bf209c048ff3b717
I20260620 20:16:38.470639 24802 ts_tablet_manager.cc:1939] T eda5a33f39884e50bf209c048ff3b717 P 33fb52c1b00347a8ac5db344e2959cac: Deleting consensus metadata
I20260620 20:16:38.470765 24193 catalog_manager.cc:4841] TS 5280c2f4f3ef4c77890e5b5b7c7f78ae (127.16.217.1:34627): tablet eda5a33f39884e50bf209c048ff3b717 (table test-table [id=6134826cd44a419cb9d5fb47a9d6b4fe]) successfully deleted
I20260620 20:16:38.471648 24193 catalog_manager.cc:4841] TS 33fb52c1b00347a8ac5db344e2959cac (127.16.217.2:41479): tablet eda5a33f39884e50bf209c048ff3b717 (table test-table [id=6134826cd44a419cb9d5fb47a9d6b4fe]) successfully deleted
I20260620 20:16:38.471993 24801 ts_tablet_manager.cc:1918] T eda5a33f39884e50bf209c048ff3b717 P b6d268e1e2e944af9e30e4b3e4b3e359: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 20:16:38.472128 24801 log.cc:1196] T eda5a33f39884e50bf209c048ff3b717 P b6d268e1e2e944af9e30e4b3e4b3e359: Deleting WAL directory at /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986536687152-17252-0/minicluster-data/ts-2/wal/wals/eda5a33f39884e50bf209c048ff3b717
I20260620 20:16:38.472584 24801 ts_tablet_manager.cc:1939] T eda5a33f39884e50bf209c048ff3b717 P b6d268e1e2e944af9e30e4b3e4b3e359: Deleting consensus metadata
I20260620 20:16:38.473635 24192 catalog_manager.cc:4841] TS b6d268e1e2e944af9e30e4b3e4b3e359 (127.16.217.3:46165): tablet eda5a33f39884e50bf209c048ff3b717 (table test-table [id=6134826cd44a419cb9d5fb47a9d6b4fe]) successfully deleted
I20260620 20:16:38.486231 24768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:38.450899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35988 (local address 127.16.217.1:34627)
0620 20:16:38.451077 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0620 20:16:38.451084 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:16:38.463913 (+ 12829us) server_negotiation.cc:407] Connection header received
0620 20:16:38.464145 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:16:38.464152 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:16:38.464238 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:16:38.464346 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:16:38.470205 (+ 5859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:38.471368 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:38.473023 (+ 1655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:38.473549 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:38.473671 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:38.479987 (+ 6316us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:16:38.480015 (+ 28us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:16:38.480032 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:16:38.480066 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:16:38.483331 (+ 3265us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:16:38.484062 (+ 731us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:16:38.484076 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:16:38.484079 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:16:38.484253 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:16:38.484651 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:16:38.484656 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:16:38.484659 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:16:38.485013 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:16:38.485319 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0620 20:16:38.485631 (+ 312us) server_negotiation.cc:299] Negotiation successful
0620 20:16:38.485839 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 20:16:38.486902 24381 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:16:38.591887 17252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu with pid 24243
I20260620 20:16:38.610648 17252 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu pid 24398
I20260620 20:16:38.624554 24771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:38.611366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48694 (local address 127.16.217.2:41479)
0620 20:16:38.611566 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0620 20:16:38.611573 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:16:38.611669 (+ 96us) server_negotiation.cc:407] Connection header received
0620 20:16:38.611878 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:16:38.611886 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:16:38.611971 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:16:38.612112 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:16:38.613402 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:38.614198 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:38.615552 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:38.615947 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:38.616117 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:38.618876 (+ 2759us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:16:38.618890 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:16:38.618908 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:16:38.618939 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:16:38.622097 (+ 3158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:16:38.622797 (+ 700us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:16:38.622804 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:16:38.622806 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:16:38.622919 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:16:38.623256 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:16:38.623259 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:16:38.623261 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:16:38.623582 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:16:38.623840 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0620 20:16:38.624044 (+ 204us) server_negotiation.cc:299] Negotiation successful
0620 20:16:38.624197 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 20:16:38.625231 24528 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:16:38.718101 17252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu with pid 24398
I20260620 20:16:38.736766 17252 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu pid 24546
I20260620 20:16:38.746353 24752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:38.745971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:36319 (local address 127.0.0.1:50798)
0620 20:16:38.746131 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0620 20:16:38.746204 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.217.62:36319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260620 20:16:38.747532 24752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:38.747211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:36319 (local address 127.0.0.1:50810)
0620 20:16:38.747346 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0620 20:16:38.747422 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.217.62:36319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260620 20:16:38.750237 24769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:38.737515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53936 (local address 127.16.217.3:46165)
0620 20:16:38.737670 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0620 20:16:38.737675 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:16:38.737694 (+ 19us) server_negotiation.cc:407] Connection header received
0620 20:16:38.737844 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:16:38.737848 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:16:38.737963 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:16:38.738064 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:16:38.739338 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:38.740066 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:38.741189 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:38.741574 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:38.741666 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:38.744122 (+ 2456us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:16:38.744134 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:16:38.744152 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:16:38.744189 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:16:38.747429 (+ 3240us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:16:38.748128 (+ 699us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:16:38.748144 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:16:38.748147 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:16:38.748262 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:16:38.748633 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:16:38.748639 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:16:38.748643 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:16:38.748974 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:16:38.749271 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0620 20:16:38.749594 (+ 323us) server_negotiation.cc:299] Negotiation successful
0620 20:16:38.749798 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 20:16:38.750854 24670 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:16:38.866990 17252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu with pid 24546
I20260620 20:16:38.888435 17252 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu pid 23876
I20260620 20:16:38.889487 24045 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:16:39.006259 17252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu with pid 23876
I20260620 20:16:39.027173 17252 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu pid 23978
I20260620 20:16:39.028291 24236 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:16:39.147006 17252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu with pid 23978
2026-06-20T20:16:39Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (9655 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986536687152-17252-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 20:16:39 dist-test-slave-2hpg krb5kdc[24814](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 20:16:39 dist-test-slave-2hpg krb5kdc[24814](info): set up 2 sockets
Jun 20 20:16:39 dist-test-slave-2hpg krb5kdc[24814](info): commencing operation
krb5kdc: starting...
W20260620 20:16:40.251060 17252 mini_kdc.cc:121] Time spent starting KDC: real 1.056s user 0.005s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986536687152-17252-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986536687152-17252-0/krb5kdc/test-user.keytab.
Jun 20 20:16:40 dist-test-slave-2hpg krb5kdc[24814](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986600, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T20:16:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T20:16:40Z Disabled control of system clock
WARNING: no policy specified for oryx/127.16.217.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.16.217.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.16.217.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986536687152-17252-0/krb5kdc/oryx_127.16.217.62.keytab.
Entry for principal oryx/127.16.217.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986536687152-17252-0/krb5kdc/oryx_127.16.217.62.keytab.
I20260620 20:16:40.455384 17252 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
/tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986536687152-17252-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986536687152-17252-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986536687152-17252-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986536687152-17252-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.217.62:39789
--webserver_interface=127.16.217.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986536687152-17252-0/krb5kdc/oryx_127.16.217.62.keytab
--principal=oryx/127.16.217.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.217.20:43657
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.217.62:39789
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986536687152-17252-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986536687152-17252-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986536687152-17252-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:16:40.774462 24827 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:16:40.774746 24827 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:16:40.774880 24827 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:16:40.782065 24827 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:16:40.782164 24827 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260620 20:16:40.782207 24827 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:16:40.782269 24827 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:16:40.782320 24827 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:16:40.782371 24827 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:16:40.792151 24827 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.217.20:43657
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986536687152-17252-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986536687152-17252-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.217.62:39789
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986536687152-17252-0/krb5kdc/oryx_127.16.217.62.keytab
--principal=oryx/127.16.217.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.217.62:39789
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986536687152-17252-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.217.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986536687152-17252-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:16:40.793344 24827 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:16:40.794740 24827 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:16:40.806146 24835 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:16:40.806192 24832 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:16:40.806394 24833 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:16:40.807240 24827 server_base.cc:1034] running on GCE node
I20260620 20:16:40.807991 24827 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:16:40.809353 24827 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:16:40.810601 24827 hybrid_clock.cc:648] HybridClock initialized: now 1781986600810536 us; error 39 us; skew 500 ppm
I20260620 20:16:40.810976 24827 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:16:40 dist-test-slave-2hpg krb5kdc[24814](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986600, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.217.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:16:40.814111 24827 init.cc:377] Logged in from keytab as oryx/127.16.217.62@KRBTEST.COM (short username oryx)
I20260620 20:16:40.815532 24827 webserver.cc:460] Webserver started at http://127.16.217.62:42197/ using document root <none> and password file <none>
I20260620 20:16:40.816179 24827 fs_manager.cc:359] Metadata directory not provided
I20260620 20:16:40.816260 24827 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:16:40.816516 24827 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:16:40.819416 24827 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986536687152-17252-0/minicluster-data/master-0/data/instance:
uuid: "827ffa1470674fdfa9e771771e6a7d94"
format_stamp: "Formatted at 2026-06-20 20:16:40 on dist-test-slave-2hpg"
server_key: "073c5044c04353ef74e73e93ee40154c"
server_key_iv: "5ffb310d4e55ae3f0659188a1f009242"
server_key_version: "encryptionkey@0"
I20260620 20:16:40.820155 24827 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986536687152-17252-0/minicluster-data/master-0/wal/instance:
uuid: "827ffa1470674fdfa9e771771e6a7d94"
format_stamp: "Formatted at 2026-06-20 20:16:40 on dist-test-slave-2hpg"
server_key: "073c5044c04353ef74e73e93ee40154c"
server_key_iv: "5ffb310d4e55ae3f0659188a1f009242"
server_key_version: "encryptionkey@0"
I20260620 20:16:40.825106 24827 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.005s
I20260620 20:16:40.828388 24842 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:16:40.829628 24827 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 20:16:40.829809 24827 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986536687152-17252-0/minicluster-data/master-0/data,/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986536687152-17252-0/minicluster-data/master-0/wal
uuid: "827ffa1470674fdfa9e771771e6a7d94"
format_stamp: "Formatted at 2026-06-20 20:16:40 on dist-test-slave-2hpg"
server_key: "073c5044c04353ef74e73e93ee40154c"
server_key_iv: "5ffb310d4e55ae3f0659188a1f009242"
server_key_version: "encryptionkey@0"
I20260620 20:16:40.830058 24827 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986536687152-17252-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986536687152-17252-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986536687152-17252-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:16:40.867250 24827 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:16:40.871742 24827 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:16:40.872025 24827 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:16:40.893726 24827 rpc_server.cc:307] RPC server started. Bound to: 127.16.217.62:39789
I20260620 20:16:40.893738 24904 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.217.62:39789 every 8 connection(s)
I20260620 20:16:40.895119 24827 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986536687152-17252-0/minicluster-data/master-0/data/info.pb
I20260620 20:16:40.895655 17252 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu as pid 24827
I20260620 20:16:40.895902 17252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986536687152-17252-0/minicluster-data/master-0/wal/instance
I20260620 20:16:40.896652 17252 external_mini_cluster.cc:1428] Setting key 2d167a6eea6979c55ecd14b9c46a3f66
I20260620 20:16:40.899724 24905 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 20 20:16:40 dist-test-slave-2hpg krb5kdc[24814](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986600, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.16.217.62@KRBTEST.COM
I20260620 20:16:40.912740 24905 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 827ffa1470674fdfa9e771771e6a7d94: Bootstrap starting.
I20260620 20:16:40.917155 24905 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 827ffa1470674fdfa9e771771e6a7d94: Neither blocks nor log segments found. Creating new log.
I20260620 20:16:40.918532 24905 log.cc:824] T 00000000000000000000000000000000 P 827ffa1470674fdfa9e771771e6a7d94: Log is configured to *not* fsync() on all Append() calls
I20260620 20:16:40.921762 24908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:40.898738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40546 (local address 127.16.217.62:39789)
0620 20:16:40.899517 (+ 779us) server_negotiation.cc:206] Beginning negotiation
0620 20:16:40.899534 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 20:16:40.899583 (+ 49us) server_negotiation.cc:407] Connection header received
0620 20:16:40.901889 (+ 2306us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:16:40.901916 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:16:40.902376 (+ 460us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:16:40.902897 (+ 521us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:16:40.904152 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:40.906390 (+ 2238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:40.907736 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:40.908805 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:40.909012 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:40.912812 (+ 3800us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:16:40.912826 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:16:40.912846 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:16:40.912901 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:16:40.917000 (+ 4099us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:16:40.917702 (+ 702us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:16:40.917711 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:16:40.917719 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:16:40.917945 (+ 226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:16:40.918320 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:16:40.918328 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:16:40.918334 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:16:40.919060 (+ 726us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:16:40.919391 (+ 331us) server_negotiation.cc:1035] Waiting for connection context
0620 20:16:40.919912 (+ 521us) server_negotiation.cc:299] Negotiation successful
0620 20:16:40.920375 (+ 463us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":556,"thread_start_us":340,"threads_started":1}
I20260620 20:16:40.922109 24905 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 827ffa1470674fdfa9e771771e6a7d94: No bootstrap required, opened a new log
I20260620 20:16:40.930392 24905 raft_consensus.cc:348] T 00000000000000000000000000000000 P 827ffa1470674fdfa9e771771e6a7d94 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "827ffa1470674fdfa9e771771e6a7d94" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 39789 } }
I20260620 20:16:40.930764 24905 raft_consensus.cc:374] T 00000000000000000000000000000000 P 827ffa1470674fdfa9e771771e6a7d94 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:16:40.930850 24905 raft_consensus.cc:729] T 00000000000000000000000000000000 P 827ffa1470674fdfa9e771771e6a7d94 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 827ffa1470674fdfa9e771771e6a7d94, State: Initialized, Role: FOLLOWER
I20260620 20:16:40.931428 24905 consensus_queue.cc:260] T 00000000000000000000000000000000 P 827ffa1470674fdfa9e771771e6a7d94 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "827ffa1470674fdfa9e771771e6a7d94" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 39789 } }
I20260620 20:16:40.931638 24905 raft_consensus.cc:388] T 00000000000000000000000000000000 P 827ffa1470674fdfa9e771771e6a7d94 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 20:16:40.931743 24905 raft_consensus.cc:482] T 00000000000000000000000000000000 P 827ffa1470674fdfa9e771771e6a7d94 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 20:16:40.931883 24905 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 827ffa1470674fdfa9e771771e6a7d94 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:16:40.934677 24905 raft_consensus.cc:504] T 00000000000000000000000000000000 P 827ffa1470674fdfa9e771771e6a7d94 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "827ffa1470674fdfa9e771771e6a7d94" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 39789 } }
I20260620 20:16:40.935142 24905 leader_election.cc:302] T 00000000000000000000000000000000 P 827ffa1470674fdfa9e771771e6a7d94 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 827ffa1470674fdfa9e771771e6a7d94; no voters:
I20260620 20:16:40.935870 24905 leader_election.cc:288] T 00000000000000000000000000000000 P 827ffa1470674fdfa9e771771e6a7d94 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 20:16:40.935971 24910 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 827ffa1470674fdfa9e771771e6a7d94 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:16:40.936786 24910 raft_consensus.cc:686] T 00000000000000000000000000000000 P 827ffa1470674fdfa9e771771e6a7d94 [term 1 LEADER]: Becoming Leader. State: Replica: 827ffa1470674fdfa9e771771e6a7d94, State: Running, Role: LEADER
I20260620 20:16:40.937399 24910 consensus_queue.cc:237] T 00000000000000000000000000000000 P 827ffa1470674fdfa9e771771e6a7d94 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "827ffa1470674fdfa9e771771e6a7d94" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 39789 } }
I20260620 20:16:40.938236 24905 sys_catalog.cc:564] T 00000000000000000000000000000000 P 827ffa1470674fdfa9e771771e6a7d94 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:16:40.940203 24912 sys_catalog.cc:455] T 00000000000000000000000000000000 P 827ffa1470674fdfa9e771771e6a7d94 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 827ffa1470674fdfa9e771771e6a7d94. Latest consensus state: current_term: 1 leader_uuid: "827ffa1470674fdfa9e771771e6a7d94" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "827ffa1470674fdfa9e771771e6a7d94" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 39789 } } }
I20260620 20:16:40.940268 24911 sys_catalog.cc:455] T 00000000000000000000000000000000 P 827ffa1470674fdfa9e771771e6a7d94 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "827ffa1470674fdfa9e771771e6a7d94" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "827ffa1470674fdfa9e771771e6a7d94" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 39789 } } }
I20260620 20:16:40.940848 24912 sys_catalog.cc:458] T 00000000000000000000000000000000 P 827ffa1470674fdfa9e771771e6a7d94 [sys.catalog]: This master's current role is: LEADER
I20260620 20:16:40.940848 24911 sys_catalog.cc:458] T 00000000000000000000000000000000 P 827ffa1470674fdfa9e771771e6a7d94 [sys.catalog]: This master's current role is: LEADER
I20260620 20:16:40.942482 24918 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 20:16:40.947909 24918 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 20:16:40.956859 24918 catalog_manager.cc:1347] Generated new cluster ID: ca0542dfe57c4c9081beb28571220a6b
I20260620 20:16:40.956974 24918 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 20:16:40.975725 24918 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 20:16:40.977669 24918 catalog_manager.cc:1504] Loading token signing keys...
I20260620 20:16:40.990155 24918 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 827ffa1470674fdfa9e771771e6a7d94: Generated new TSK 0
I20260620 20:16:40.991258 24918 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.16.217.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.16.217.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.16.217.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986536687152-17252-0/krb5kdc/oryx_127.16.217.1.keytab.
Entry for principal oryx/127.16.217.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986536687152-17252-0/krb5kdc/oryx_127.16.217.1.keytab.
I20260620 20:16:41.047219 17252 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
/tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986536687152-17252-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986536687152-17252-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986536687152-17252-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986536687152-17252-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.217.1:0
--local_ip_for_outbound_sockets=127.16.217.1
--webserver_interface=127.16.217.1
--webserver_port=0
--tserver_master_addrs=127.16.217.62:39789
--keytab_file=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986536687152-17252-0/krb5kdc/oryx_127.16.217.1.keytab
--principal=oryx/127.16.217.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.217.20:43657
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986536687152-17252-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986536687152-17252-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986536687152-17252-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:16:41.379556 24931 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:16:41.379832 24931 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:16:41.379992 24931 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:16:41.387329 24931 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:16:41.387439 24931 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:16:41.387581 24931 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 20:16:41.387712 24931 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.217.1
I20260620 20:16:41.397571 24931 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.217.20:43657
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986536687152-17252-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986536687152-17252-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986536687152-17252-0/krb5kdc/oryx_127.16.217.1.keytab
--principal=oryx/127.16.217.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.217.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986536687152-17252-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.217.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.16.217.62:39789
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.217.1
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986536687152-17252-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:16:41.398876 24931 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:16:41.400208 24931 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:16:41.413237 24936 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:16:41.413290 24937 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:16:41.413257 24939 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:16:41.414637 24931 server_base.cc:1034] running on GCE node
I20260620 20:16:41.415323 24931 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:16:41.416288 24931 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:16:41.417507 24931 hybrid_clock.cc:648] HybridClock initialized: now 1781986601417424 us; error 53 us; skew 500 ppm
I20260620 20:16:41.417913 24931 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:16:41 dist-test-slave-2hpg krb5kdc[24814](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986601, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.217.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:16:41.421397 24931 init.cc:377] Logged in from keytab as oryx/127.16.217.1@KRBTEST.COM (short username oryx)
I20260620 20:16:41.422912 24931 webserver.cc:460] Webserver started at http://127.16.217.1:33151/ using document root <none> and password file <none>
I20260620 20:16:41.423745 24931 fs_manager.cc:359] Metadata directory not provided
I20260620 20:16:41.423879 24931 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:16:41.424208 24931 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:16:41.427199 24931 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986536687152-17252-0/minicluster-data/ts-0/data/instance:
uuid: "5a07c01f0567489fb51b6bb676a3764f"
format_stamp: "Formatted at 2026-06-20 20:16:41 on dist-test-slave-2hpg"
server_key: "241ff954dedcca0bec570dd79691f196"
server_key_iv: "3e144f13c9e06e827f57abdc83f51c7d"
server_key_version: "encryptionkey@0"
I20260620 20:16:41.427994 24931 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986536687152-17252-0/minicluster-data/ts-0/wal/instance:
uuid: "5a07c01f0567489fb51b6bb676a3764f"
format_stamp: "Formatted at 2026-06-20 20:16:41 on dist-test-slave-2hpg"
server_key: "241ff954dedcca0bec570dd79691f196"
server_key_iv: "3e144f13c9e06e827f57abdc83f51c7d"
server_key_version: "encryptionkey@0"
I20260620 20:16:41.432977 24931 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 20:16:41.436489 24946 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:16:41.437950 24931 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260620 20:16:41.438131 24931 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986536687152-17252-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986536687152-17252-0/minicluster-data/ts-0/wal
uuid: "5a07c01f0567489fb51b6bb676a3764f"
format_stamp: "Formatted at 2026-06-20 20:16:41 on dist-test-slave-2hpg"
server_key: "241ff954dedcca0bec570dd79691f196"
server_key_iv: "3e144f13c9e06e827f57abdc83f51c7d"
server_key_version: "encryptionkey@0"
I20260620 20:16:41.438362 24931 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986536687152-17252-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986536687152-17252-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986536687152-17252-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:16:41.479261 24931 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:16:41.483873 24931 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:16:41.484192 24931 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:16:41.487192 24931 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:16:41.487325 24931 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:16:41.487567 24931 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:16:41.487658 24931 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 20:16:41 dist-test-slave-2hpg krb5kdc[24814](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986601, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.217.1@KRBTEST.COM for oryx/127.16.217.62@KRBTEST.COM
I20260620 20:16:41.512148 24955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:41.487593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.217.1:46005 (local address 127.16.217.62:39789)
0620 20:16:41.488131 (+ 538us) server_negotiation.cc:206] Beginning negotiation
0620 20:16:41.488137 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:16:41.489349 (+ 1212us) server_negotiation.cc:407] Connection header received
0620 20:16:41.491448 (+ 2099us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:16:41.491458 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:16:41.491570 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:16:41.491721 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:16:41.494038 (+ 2317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:41.495036 (+ 998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:41.497562 (+ 2526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:41.498066 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:41.498215 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:41.503189 (+ 4974us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:16:41.503212 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:16:41.503221 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:16:41.503282 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:16:41.507767 (+ 4485us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:16:41.508741 (+ 974us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:16:41.508751 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:16:41.508758 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:16:41.508979 (+ 221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:16:41.509536 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:16:41.509544 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:16:41.509551 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:16:41.510058 (+ 507us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:16:41.510935 (+ 877us) server_negotiation.cc:1035] Waiting for connection context
0620 20:16:41.511534 (+ 599us) server_negotiation.cc:299] Negotiation successful
0620 20:16:41.511780 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":414,"thread_start_us":303,"threads_started":1}
I20260620 20:16:41.513269 24957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:41.488289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:39789 (local address 127.16.217.1:46005)
0620 20:16:41.489151 (+ 862us) negotiation.cc:107] Waiting for socket to connect
0620 20:16:41.489206 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0620 20:16:41.491147 (+ 1941us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:16:41.491980 (+ 833us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:16:41.491988 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:16:41.492766 (+ 778us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:16:41.493815 (+ 1049us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:16:41.493853 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:16:41.495182 (+ 1329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:16:41.495187 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:16:41.497407 (+ 2220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:16:41.497420 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:16:41.498338 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:16:41.498345 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:16:41.498708 (+ 363us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:41.499631 (+ 923us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:16:41.499653 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:16:41.502870 (+ 3217us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:16:41.507999 (+ 5129us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:16:41.508012 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:16:41.508037 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:16:41.508572 (+ 535us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:16:41.509135 (+ 563us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:16:41.509139 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:16:41.509142 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:16:41.509392 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:16:41.510258 (+ 866us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:16:41.510270 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:16:41.511006 (+ 736us) client_negotiation.cc:769] Sending connection context
0620 20:16:41.511420 (+ 414us) client_negotiation.cc:240] Negotiation successful
0620 20:16:41.512029 (+ 609us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":589,"thread_start_us":380,"threads_started":1}
I20260620 20:16:41.538221 25067 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.217.1:43535 every 8 connection(s)
I20260620 20:16:41.538295 24931 rpc_server.cc:307] RPC server started. Bound to: 127.16.217.1:43535
I20260620 20:16:41.540273 24931 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986536687152-17252-0/minicluster-data/ts-0/data/info.pb
I20260620 20:16:41.540616 25029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:41.521014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:39789 (local address 127.16.217.1:53623)
0620 20:16:41.522392 (+ 1378us) negotiation.cc:107] Waiting for socket to connect
0620 20:16:41.522420 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0620 20:16:41.522607 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:16:41.523035 (+ 428us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:16:41.523044 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:16:41.524050 (+ 1006us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:16:41.524845 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:16:41.524872 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:16:41.526611 (+ 1739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:16:41.526621 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:16:41.528900 (+ 2279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:16:41.528921 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:16:41.529811 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:16:41.529826 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:16:41.530120 (+ 294us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:41.531122 (+ 1002us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:16:41.531159 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:16:41.533037 (+ 1878us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:16:41.537442 (+ 4405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:16:41.537451 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:16:41.537457 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:16:41.538271 (+ 814us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:16:41.538733 (+ 462us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:16:41.538738 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:16:41.538744 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:16:41.538903 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:16:41.539553 (+ 650us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:16:41.539557 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:16:41.539784 (+ 227us) client_negotiation.cc:769] Sending connection context
0620 20:16:41.539918 (+ 134us) client_negotiation.cc:240] Negotiation successful
0620 20:16:41.540174 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1231,"thread_start_us":1061,"threads_started":1}
I20260620 20:16:41.540748 24955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:41.520977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.217.1:53623 (local address 127.16.217.62:39789)
0620 20:16:41.521210 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0620 20:16:41.521219 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 20:16:41.522526 (+ 1307us) server_negotiation.cc:407] Connection header received
0620 20:16:41.522747 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:16:41.522755 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:16:41.522856 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:16:41.522965 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:16:41.525061 (+ 2096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:41.526392 (+ 1331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:41.529104 (+ 2712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:41.529560 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:41.529674 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:41.533344 (+ 3670us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:16:41.533352 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:16:41.533358 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:16:41.533418 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:16:41.537237 (+ 3819us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:16:41.538419 (+ 1182us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:16:41.538427 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:16:41.538433 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:16:41.538594 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:16:41.539022 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:16:41.539026 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:16:41.539028 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:16:41.539310 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:16:41.539495 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0620 20:16:41.540084 (+ 589us) server_negotiation.cc:299] Negotiation successful
0620 20:16:41.540358 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260620 20:16:41.542867 25068 heartbeater.cc:347] Connected to a master server at 127.16.217.62:39789
I20260620 20:16:41.543136 25068 heartbeater.cc:464] Registering TS with master...
I20260620 20:16:41.543855 25068 heartbeater.cc:511] Master 127.16.217.62:39789 requested a full tablet report, sending...
I20260620 20:16:41.545842 24860 ts_manager.cc:194] Registered new tserver with Master: 5a07c01f0567489fb51b6bb676a3764f (127.16.217.1:43535)
I20260620 20:16:41.547372 24860 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.16.217.1@KRBTEST.COM'} at 127.16.217.1:46005
I20260620 20:16:41.549299 17252 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu as pid 24931
I20260620 20:16:41.549489 17252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986536687152-17252-0/minicluster-data/ts-0/wal/instance
I20260620 20:16:41.550176 17252 external_mini_cluster.cc:1428] Setting key 0e35d37ef4f6e021c67d27fdbcbbdbbc
WARNING: no policy specified for oryx/127.16.217.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.16.217.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.16.217.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986536687152-17252-0/krb5kdc/oryx_127.16.217.2.keytab.
Entry for principal oryx/127.16.217.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986536687152-17252-0/krb5kdc/oryx_127.16.217.2.keytab.
I20260620 20:16:41.594751 17252 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
/tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986536687152-17252-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986536687152-17252-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986536687152-17252-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986536687152-17252-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.217.2:0
--local_ip_for_outbound_sockets=127.16.217.2
--webserver_interface=127.16.217.2
--webserver_port=0
--tserver_master_addrs=127.16.217.62:39789
--keytab_file=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986536687152-17252-0/krb5kdc/oryx_127.16.217.2.keytab
--principal=oryx/127.16.217.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.217.20:43657
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986536687152-17252-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986536687152-17252-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986536687152-17252-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:16:41.917137 25072 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:16:41.917392 25072 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:16:41.917553 25072 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:16:41.924911 25072 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:16:41.925022 25072 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:16:41.925166 25072 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 20:16:41.925282 25072 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.217.2
I20260620 20:16:41.935312 25072 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.217.20:43657
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986536687152-17252-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986536687152-17252-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986536687152-17252-0/krb5kdc/oryx_127.16.217.2.keytab
--principal=oryx/127.16.217.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.217.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986536687152-17252-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.217.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.16.217.62:39789
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.217.2
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986536687152-17252-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:16:41.936618 25072 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:16:41.938190 25072 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:16:41.951855 25081 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:16:41.951855 25077 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:16:41.951892 25078 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:16:41.952806 25072 server_base.cc:1034] running on GCE node
I20260620 20:16:41.953433 25072 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:16:41.954484 25072 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:16:41.955796 25072 hybrid_clock.cc:648] HybridClock initialized: now 1781986601955737 us; error 66 us; skew 500 ppm
I20260620 20:16:41.956110 25072 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:16:41 dist-test-slave-2hpg krb5kdc[24814](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986601, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.217.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:16:41.959304 25072 init.cc:377] Logged in from keytab as oryx/127.16.217.2@KRBTEST.COM (short username oryx)
I20260620 20:16:41.960901 25072 webserver.cc:460] Webserver started at http://127.16.217.2:35585/ using document root <none> and password file <none>
I20260620 20:16:41.961550 25072 fs_manager.cc:359] Metadata directory not provided
I20260620 20:16:41.961634 25072 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:16:41.961978 25072 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:16:41.964715 25072 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986536687152-17252-0/minicluster-data/ts-1/data/instance:
uuid: "a64d4977929340b7a5de6108995a78fa"
format_stamp: "Formatted at 2026-06-20 20:16:41 on dist-test-slave-2hpg"
server_key: "59eb6aea5382c3d9303239f67d60d740"
server_key_iv: "28106b14cd6fa18889a7f2ce4dd0f87d"
server_key_version: "encryptionkey@0"
I20260620 20:16:41.965472 25072 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986536687152-17252-0/minicluster-data/ts-1/wal/instance:
uuid: "a64d4977929340b7a5de6108995a78fa"
format_stamp: "Formatted at 2026-06-20 20:16:41 on dist-test-slave-2hpg"
server_key: "59eb6aea5382c3d9303239f67d60d740"
server_key_iv: "28106b14cd6fa18889a7f2ce4dd0f87d"
server_key_version: "encryptionkey@0"
I20260620 20:16:41.970402 25072 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260620 20:16:41.973730 25088 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:16:41.975158 25072 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 20:16:41.975286 25072 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986536687152-17252-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986536687152-17252-0/minicluster-data/ts-1/wal
uuid: "a64d4977929340b7a5de6108995a78fa"
format_stamp: "Formatted at 2026-06-20 20:16:41 on dist-test-slave-2hpg"
server_key: "59eb6aea5382c3d9303239f67d60d740"
server_key_iv: "28106b14cd6fa18889a7f2ce4dd0f87d"
server_key_version: "encryptionkey@0"
I20260620 20:16:41.975468 25072 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986536687152-17252-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986536687152-17252-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986536687152-17252-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:16:41.994267 25072 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:16:41.998967 25072 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:16:41.999186 25072 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:16:42.001672 25072 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:16:42.001756 25072 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:16:42.001945 25072 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:16:42.002019 25072 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 20:16:42 dist-test-slave-2hpg krb5kdc[24814](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986601, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.217.2@KRBTEST.COM for oryx/127.16.217.62@KRBTEST.COM
I20260620 20:16:42.028239 24955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:42.002696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.217.2:56093 (local address 127.16.217.62:39789)
0620 20:16:42.002889 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0620 20:16:42.002894 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:16:42.004623 (+ 1729us) server_negotiation.cc:407] Connection header received
0620 20:16:42.006975 (+ 2352us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:16:42.006980 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:16:42.007071 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:16:42.007192 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:16:42.009992 (+ 2800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:42.011296 (+ 1304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:42.013532 (+ 2236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:42.014405 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:42.014561 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:42.020470 (+ 5909us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:16:42.020477 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:16:42.020482 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:16:42.020524 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:16:42.024571 (+ 4047us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:16:42.025662 (+ 1091us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:16:42.025678 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:16:42.025683 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:16:42.025817 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:16:42.026355 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:16:42.026360 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:16:42.026365 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:16:42.026645 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:16:42.026824 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0620 20:16:42.027549 (+ 725us) server_negotiation.cc:299] Negotiation successful
0620 20:16:42.027785 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260620 20:16:42.029731 25097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:42.003614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:39789 (local address 127.16.217.2:56093)
0620 20:16:42.004428 (+ 814us) negotiation.cc:107] Waiting for socket to connect
0620 20:16:42.004466 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 20:16:42.006654 (+ 2188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:16:42.007456 (+ 802us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:16:42.007471 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:16:42.008638 (+ 1167us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:16:42.009739 (+ 1101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:16:42.009763 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:16:42.011487 (+ 1724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:16:42.011493 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:16:42.013341 (+ 1848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:16:42.013360 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:16:42.014957 (+ 1597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:16:42.014966 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:16:42.015388 (+ 422us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:42.016593 (+ 1205us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:16:42.016631 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:16:42.020215 (+ 3584us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:16:42.024763 (+ 4548us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:16:42.024777 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:16:42.024790 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:16:42.025512 (+ 722us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:16:42.025987 (+ 475us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:16:42.025992 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:16:42.025996 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:16:42.026224 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:16:42.026800 (+ 576us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:16:42.026809 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:16:42.027176 (+ 367us) client_negotiation.cc:769] Sending connection context
0620 20:16:42.027457 (+ 281us) client_negotiation.cc:240] Negotiation successful
0620 20:16:42.027994 (+ 537us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":572,"thread_start_us":331,"threads_started":1}
I20260620 20:16:42.053845 25207 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.217.2:36825 every 8 connection(s)
I20260620 20:16:42.053900 25072 rpc_server.cc:307] RPC server started. Bound to: 127.16.217.2:36825
I20260620 20:16:42.055609 25072 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986536687152-17252-0/minicluster-data/ts-1/data/info.pb
I20260620 20:16:42.057812 25208 heartbeater.cc:347] Connected to a master server at 127.16.217.62:39789
I20260620 20:16:42.058075 25208 heartbeater.cc:464] Registering TS with master...
I20260620 20:16:42.058907 25190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:42.041816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:39789 (local address 127.16.217.2:50169)
0620 20:16:42.042782 (+ 966us) negotiation.cc:107] Waiting for socket to connect
0620 20:16:42.042799 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 20:16:42.043023 (+ 224us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:16:42.043637 (+ 614us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:16:42.043644 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:16:42.044409 (+ 765us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:16:42.044967 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:16:42.044992 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:16:42.046767 (+ 1775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:16:42.046771 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:16:42.047911 (+ 1140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:16:42.047923 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:16:42.049260 (+ 1337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:16:42.049267 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:16:42.049551 (+ 284us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:42.050522 (+ 971us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:16:42.050541 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:16:42.051911 (+ 1370us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:16:42.055920 (+ 4009us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:16:42.055924 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:16:42.055927 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:16:42.056483 (+ 556us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:16:42.057018 (+ 535us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:16:42.057022 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:16:42.057026 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:16:42.057193 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:16:42.057958 (+ 765us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:16:42.057963 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:16:42.058168 (+ 205us) client_negotiation.cc:769] Sending connection context
0620 20:16:42.058308 (+ 140us) client_negotiation.cc:240] Negotiation successful
0620 20:16:42.058566 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":757,"thread_start_us":580,"threads_started":1}
I20260620 20:16:42.058972 24955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:42.041737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.217.2:50169 (local address 127.16.217.62:39789)
0620 20:16:42.041986 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0620 20:16:42.041996 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 20:16:42.042905 (+ 909us) server_negotiation.cc:407] Connection header received
0620 20:16:42.043211 (+ 306us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:16:42.043220 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:16:42.043314 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:16:42.043436 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:16:42.045176 (+ 1740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:42.046621 (+ 1445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:42.048079 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:42.049007 (+ 928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:42.049167 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:42.052126 (+ 2959us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:16:42.052132 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:16:42.052135 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:16:42.052168 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:16:42.055799 (+ 3631us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:16:42.056647 (+ 848us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:16:42.056653 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:16:42.056656 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:16:42.056867 (+ 211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:16:42.057324 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:16:42.057328 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:16:42.057331 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:16:42.057817 (+ 486us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:16:42.057997 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0620 20:16:42.058422 (+ 425us) server_negotiation.cc:299] Negotiation successful
0620 20:16:42.058636 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260620 20:16:42.058977 25208 heartbeater.cc:511] Master 127.16.217.62:39789 requested a full tablet report, sending...
I20260620 20:16:42.060453 24859 ts_manager.cc:194] Registered new tserver with Master: a64d4977929340b7a5de6108995a78fa (127.16.217.2:36825)
I20260620 20:16:42.061437 24859 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.16.217.2@KRBTEST.COM'} at 127.16.217.2:56093
I20260620 20:16:42.065866 17252 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu as pid 25072
I20260620 20:16:42.066099 17252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986536687152-17252-0/minicluster-data/ts-1/wal/instance
I20260620 20:16:42.066525 17252 external_mini_cluster.cc:1428] Setting key 73c140c079a8e9f31a1813dc574afd6a
WARNING: no policy specified for oryx/127.16.217.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.16.217.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.16.217.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986536687152-17252-0/krb5kdc/oryx_127.16.217.3.keytab.
Entry for principal oryx/127.16.217.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986536687152-17252-0/krb5kdc/oryx_127.16.217.3.keytab.
I20260620 20:16:42.112851 17252 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
/tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986536687152-17252-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986536687152-17252-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986536687152-17252-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986536687152-17252-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.217.3:0
--local_ip_for_outbound_sockets=127.16.217.3
--webserver_interface=127.16.217.3
--webserver_port=0
--tserver_master_addrs=127.16.217.62:39789
--keytab_file=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986536687152-17252-0/krb5kdc/oryx_127.16.217.3.keytab
--principal=oryx/127.16.217.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.217.20:43657
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986536687152-17252-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986536687152-17252-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986536687152-17252-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:16:42.438036 25212 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:16:42.438310 25212 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:16:42.438563 25212 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:16:42.446626 25212 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:16:42.446748 25212 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:16:42.446827 25212 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 20:16:42.446905 25212 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.217.3
I20260620 20:16:42.456405 25212 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.217.20:43657
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986536687152-17252-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986536687152-17252-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986536687152-17252-0/krb5kdc/oryx_127.16.217.3.keytab
--principal=oryx/127.16.217.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.217.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986536687152-17252-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.217.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.16.217.62:39789
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.217.3
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986536687152-17252-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:16:42.457612 25212 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:16:42.458897 25212 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:16:42.471983 25219 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:16:42.471940 25218 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:16:42.472054 25221 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:16:42.472699 25212 server_base.cc:1034] running on GCE node
I20260620 20:16:42.473383 25212 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:16:42.474360 25212 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:16:42.475569 25212 hybrid_clock.cc:648] HybridClock initialized: now 1781986602475491 us; error 54 us; skew 500 ppm
I20260620 20:16:42.475835 25212 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:16:42 dist-test-slave-2hpg krb5kdc[24814](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986602, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.217.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:16:42.478894 25212 init.cc:377] Logged in from keytab as oryx/127.16.217.3@KRBTEST.COM (short username oryx)
I20260620 20:16:42.480502 25212 webserver.cc:460] Webserver started at http://127.16.217.3:45405/ using document root <none> and password file <none>
I20260620 20:16:42.481137 25212 fs_manager.cc:359] Metadata directory not provided
I20260620 20:16:42.481217 25212 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:16:42.481489 25212 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:16:42.484359 25212 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986536687152-17252-0/minicluster-data/ts-2/data/instance:
uuid: "386309b3999d45769c3723f68adf1210"
format_stamp: "Formatted at 2026-06-20 20:16:42 on dist-test-slave-2hpg"
server_key: "c17387179e91d9cf17d803ee90bf8b87"
server_key_iv: "370874b70314609ebb61011d78e6f2b4"
server_key_version: "encryptionkey@0"
I20260620 20:16:42.485082 25212 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986536687152-17252-0/minicluster-data/ts-2/wal/instance:
uuid: "386309b3999d45769c3723f68adf1210"
format_stamp: "Formatted at 2026-06-20 20:16:42 on dist-test-slave-2hpg"
server_key: "c17387179e91d9cf17d803ee90bf8b87"
server_key_iv: "370874b70314609ebb61011d78e6f2b4"
server_key_version: "encryptionkey@0"
I20260620 20:16:42.490136 25212 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260620 20:16:42.493592 25228 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:16:42.494990 25212 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 20:16:42.495119 25212 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986536687152-17252-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986536687152-17252-0/minicluster-data/ts-2/wal
uuid: "386309b3999d45769c3723f68adf1210"
format_stamp: "Formatted at 2026-06-20 20:16:42 on dist-test-slave-2hpg"
server_key: "c17387179e91d9cf17d803ee90bf8b87"
server_key_iv: "370874b70314609ebb61011d78e6f2b4"
server_key_version: "encryptionkey@0"
I20260620 20:16:42.495298 25212 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986536687152-17252-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986536687152-17252-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986536687152-17252-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:16:42.514892 25212 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:16:42.518970 25212 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:16:42.519209 25212 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:16:42.521736 25212 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:16:42.521891 25212 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:16:42.522048 25212 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:16:42.522117 25212 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 20:16:42 dist-test-slave-2hpg krb5kdc[24814](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986602, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.217.3@KRBTEST.COM for oryx/127.16.217.62@KRBTEST.COM
I20260620 20:16:42.548960 24955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:42.522580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.217.3:33799 (local address 127.16.217.62:39789)
0620 20:16:42.522851 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0620 20:16:42.522855 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:16:42.524288 (+ 1433us) server_negotiation.cc:407] Connection header received
0620 20:16:42.526533 (+ 2245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:16:42.526540 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:16:42.526644 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:16:42.526773 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:16:42.529111 (+ 2338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:42.530218 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:42.533548 (+ 3330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:42.534211 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:42.534347 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:42.540782 (+ 6435us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:16:42.540793 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:16:42.540801 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:16:42.540855 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:16:42.545122 (+ 4267us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:16:42.546176 (+ 1054us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:16:42.546184 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:16:42.546189 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:16:42.546380 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:16:42.546795 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:16:42.546803 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:16:42.546809 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:16:42.547270 (+ 461us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:16:42.547430 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0620 20:16:42.548100 (+ 670us) server_negotiation.cc:299] Negotiation successful
0620 20:16:42.548442 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":169}
I20260620 20:16:42.549489 25237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:42.523252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:39789 (local address 127.16.217.3:33799)
0620 20:16:42.524100 (+ 848us) negotiation.cc:107] Waiting for socket to connect
0620 20:16:42.524146 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 20:16:42.526159 (+ 2013us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:16:42.527115 (+ 956us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:16:42.527127 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:16:42.527945 (+ 818us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:16:42.528935 (+ 990us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:16:42.528959 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:16:42.530439 (+ 1480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:16:42.530448 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:16:42.533381 (+ 2933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:16:42.533402 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:16:42.534713 (+ 1311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:16:42.534721 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:16:42.535061 (+ 340us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:42.535919 (+ 858us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:16:42.535940 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:16:42.540481 (+ 4541us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:16:42.545311 (+ 4830us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:16:42.545326 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:16:42.545347 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:16:42.546006 (+ 659us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:16:42.546520 (+ 514us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:16:42.546524 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:16:42.546526 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:16:42.546683 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:16:42.547398 (+ 715us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:16:42.547404 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:16:42.547712 (+ 308us) client_negotiation.cc:769] Sending connection context
0620 20:16:42.547933 (+ 221us) client_negotiation.cc:240] Negotiation successful
0620 20:16:42.548330 (+ 397us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":606,"thread_start_us":362,"threads_started":1}
I20260620 20:16:42.550125 25068 heartbeater.cc:503] Master 127.16.217.62:39789 was elected leader, sending a full tablet report...
I20260620 20:16:42.572134 25293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:42.556809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:39789 (local address 127.16.217.3:51629)
0620 20:16:42.557834 (+ 1025us) negotiation.cc:107] Waiting for socket to connect
0620 20:16:42.557879 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 20:16:42.558037 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:16:42.558670 (+ 633us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:16:42.558674 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:16:42.559477 (+ 803us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:16:42.560309 (+ 832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:16:42.560326 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:16:42.561610 (+ 1284us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:16:42.561619 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:16:42.563159 (+ 1540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:16:42.563170 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:16:42.563959 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:16:42.563966 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:16:42.564124 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:42.564796 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:16:42.564811 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:16:42.565494 (+ 683us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:16:42.569471 (+ 3977us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:16:42.569477 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:16:42.569481 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:16:42.569945 (+ 464us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:16:42.570310 (+ 365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:16:42.570315 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:16:42.570318 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:16:42.570442 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:16:42.571135 (+ 693us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:16:42.571139 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:16:42.571358 (+ 219us) client_negotiation.cc:769] Sending connection context
0620 20:16:42.571529 (+ 171us) client_negotiation.cc:240] Negotiation successful
0620 20:16:42.571831 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":812,"thread_start_us":592,"threads_started":1}
I20260620 20:16:42.572497 24955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:42.556725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.217.3:51629 (local address 127.16.217.62:39789)
0620 20:16:42.556970 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0620 20:16:42.556977 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:16:42.557987 (+ 1010us) server_negotiation.cc:407] Connection header received
0620 20:16:42.558231 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:16:42.558236 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:16:42.558318 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:16:42.558403 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:16:42.560495 (+ 2092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:42.561444 (+ 949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:42.563318 (+ 1874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:42.563783 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:42.563920 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:42.565662 (+ 1742us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:16:42.565667 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:16:42.565670 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:16:42.565696 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:16:42.569328 (+ 3632us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:16:42.570058 (+ 730us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:16:42.570066 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:16:42.570071 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:16:42.570200 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:16:42.570554 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:16:42.570559 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:16:42.570562 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:16:42.570911 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:16:42.571070 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0620 20:16:42.571704 (+ 634us) server_negotiation.cc:299] Negotiation successful
0620 20:16:42.571998 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260620 20:16:42.580307 25212 rpc_server.cc:307] RPC server started. Bound to: 127.16.217.3:44465
I20260620 20:16:42.580348 25347 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.217.3:44465 every 8 connection(s)
I20260620 20:16:42.581490 25212 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986536687152-17252-0/minicluster-data/ts-2/data/info.pb
I20260620 20:16:42.583635 25348 heartbeater.cc:347] Connected to a master server at 127.16.217.62:39789
I20260620 20:16:42.583875 25348 heartbeater.cc:464] Registering TS with master...
I20260620 20:16:42.584353 17252 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu as pid 25212
I20260620 20:16:42.584530 17252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986536687152-17252-0/minicluster-data/ts-2/wal/instance
I20260620 20:16:42.584753 25348 heartbeater.cc:511] Master 127.16.217.62:39789 requested a full tablet report, sending...
I20260620 20:16:42.585094 17252 external_mini_cluster.cc:1428] Setting key eb59ad3db4bbf3e53df229c4ba95a1ad
I20260620 20:16:42.586437 24859 ts_manager.cc:194] Registered new tserver with Master: 386309b3999d45769c3723f68adf1210 (127.16.217.3:44465)
I20260620 20:16:42.587500 24859 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.16.217.3@KRBTEST.COM'} at 127.16.217.3:33799
I20260620 20:16:42.588171 17252 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 20:16:42 dist-test-slave-2hpg krb5kdc[24814](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986602, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 20:16:42 dist-test-slave-2hpg krb5kdc[24814](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.16.217.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 20:16:42 dist-test-slave-2hpg krb5kdc[24814](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.16.217.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 20:16:43.063966 25208 heartbeater.cc:503] Master 127.16.217.62:39789 was elected leader, sending a full tablet report...
I20260620 20:16:43.589986 25348 heartbeater.cc:503] Master 127.16.217.62:39789 was elected leader, sending a full tablet report...
W20260620 20:16:43.748692 25356 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.115s user 0.003s sys 0.011s
W20260620 20:16:43.749948 24955 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:16:42.627772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40548 (local address 127.16.217.62:39789)
0620 20:16:42.628005 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0620 20:16:42.628013 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 20:16:42.628213 (+ 200us) server_negotiation.cc:407] Connection header received
0620 20:16:42.628474 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:16:42.628479 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:16:42.628546 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:16:42.628662 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:16:42.630142 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:42.630901 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:42.632265 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:42.632810 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:42.632960 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:43.749612 (+1116652us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40548: BlockingRecv error: recv got EOF from 127.0.0.1:40548 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
Jun 20 20:16:43 dist-test-slave-2hpg krb5kdc[24814](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986602, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.16.217.62@KRBTEST.COM
I20260620 20:16:43.773072 24955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:43.755418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40564 (local address 127.16.217.62:39789)
0620 20:16:43.755624 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0620 20:16:43.755630 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:16:43.755830 (+ 200us) server_negotiation.cc:407] Connection header received
0620 20:16:43.756042 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:16:43.756051 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:16:43.756124 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:16:43.756233 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:16:43.757405 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:43.758182 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:43.759443 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:43.759885 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:43.760016 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:43.768071 (+ 8055us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:16:43.768076 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:16:43.768079 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:16:43.768116 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:16:43.770697 (+ 2581us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:16:43.771373 (+ 676us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:16:43.771377 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:16:43.771379 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:16:43.771493 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:16:43.771867 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:16:43.771870 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:16:43.771872 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:16:43.772120 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:16:43.772308 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0620 20:16:43.772632 (+ 324us) server_negotiation.cc:299] Negotiation successful
0620 20:16:43.772843 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 20:16:43.777318 24859 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40564:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 20:16:43.784085 24859 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 20:16:43.807353 25369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:43.795948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38734 (local address 127.16.217.2:36825)
0620 20:16:43.797471 (+ 1523us) server_negotiation.cc:206] Beginning negotiation
0620 20:16:43.797480 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 20:16:43.798441 (+ 961us) server_negotiation.cc:407] Connection header received
0620 20:16:43.798660 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:16:43.798669 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:16:43.798870 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:16:43.799929 (+ 1059us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:16:43.800107 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:43.801062 (+ 955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:43.804631 (+ 3569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:43.806120 (+ 1489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:43.806247 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:43.806447 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0620 20:16:43.806827 (+ 380us) server_negotiation.cc:299] Negotiation successful
0620 20:16:43.807024 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1372,"thread_start_us":1233,"threads_started":1}
I20260620 20:16:43.807902 25368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:43.795716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.2:36825 (local address 127.0.0.1:38734)
0620 20:16:43.798093 (+ 2377us) negotiation.cc:107] Waiting for socket to connect
0620 20:16:43.798152 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0620 20:16:43.798339 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:16:43.799064 (+ 725us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:16:43.799071 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:16:43.799100 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:16:43.799598 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:16:43.799611 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:16:43.801203 (+ 1592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:16:43.801209 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:16:43.804470 (+ 3261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:16:43.804483 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:16:43.806326 (+ 1843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:16:43.806336 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:16:43.806550 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:43.806604 (+ 54us) client_negotiation.cc:769] Sending connection context
0620 20:16:43.807378 (+ 774us) client_negotiation.cc:240] Negotiation successful
0620 20:16:43.807546 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2038,"mutex_wait_us":25,"thread_start_us":2202,"threads_started":1}
I20260620 20:16:43.809373 25370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:43.796456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38004 (local address 127.16.217.1:43535)
0620 20:16:43.801683 (+ 5227us) server_negotiation.cc:206] Beginning negotiation
0620 20:16:43.801703 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0620 20:16:43.801729 (+ 26us) server_negotiation.cc:407] Connection header received
0620 20:16:43.801826 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:16:43.801836 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:16:43.802104 (+ 268us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:16:43.802309 (+ 205us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:16:43.802961 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:43.804239 (+ 1278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:43.806630 (+ 2391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:43.808400 (+ 1770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:43.808548 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:43.808763 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0620 20:16:43.808968 (+ 205us) server_negotiation.cc:299] Negotiation successful
0620 20:16:43.809110 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5082,"thread_start_us":4899,"threads_started":1}
I20260620 20:16:43.809437 25367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:43.795931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.1:43535 (local address 127.0.0.1:38004)
0620 20:16:43.798248 (+ 2317us) negotiation.cc:107] Waiting for socket to connect
0620 20:16:43.798269 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 20:16:43.798349 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:16:43.802294 (+ 3945us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:16:43.802299 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:16:43.802326 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:16:43.802766 (+ 440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:16:43.802778 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:16:43.804365 (+ 1587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:16:43.804372 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:16:43.806463 (+ 2091us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:16:43.806478 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:16:43.808603 (+ 2125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:16:43.808610 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:16:43.808754 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:43.808792 (+ 38us) client_negotiation.cc:769] Sending connection context
0620 20:16:43.808931 (+ 139us) client_negotiation.cc:240] Negotiation successful
0620 20:16:43.809047 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2184,"mutex_wait_us":1,"thread_start_us":1692,"threads_started":1}
I20260620 20:16:43.810287 25372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:43.799137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.3:44465 (local address 127.0.0.1:49776)
0620 20:16:43.802059 (+ 2922us) negotiation.cc:107] Waiting for socket to connect
0620 20:16:43.802081 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 20:16:43.802183 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:16:43.802959 (+ 776us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:16:43.802963 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:16:43.802985 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:16:43.803438 (+ 453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:16:43.803452 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:16:43.805041 (+ 1589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:16:43.805047 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:16:43.807140 (+ 2093us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:16:43.807153 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:16:43.809536 (+ 2383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:16:43.809544 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:16:43.809704 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:43.809739 (+ 35us) client_negotiation.cc:769] Sending connection context
0620 20:16:43.809836 (+ 97us) client_negotiation.cc:240] Negotiation successful
0620 20:16:43.809987 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2821,"thread_start_us":2647,"threads_started":1}
I20260620 20:16:43.810356 25371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:43.798815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49776 (local address 127.16.217.3:44465)
0620 20:16:43.802181 (+ 3366us) server_negotiation.cc:206] Beginning negotiation
0620 20:16:43.802187 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:16:43.802209 (+ 22us) server_negotiation.cc:407] Connection header received
0620 20:16:43.802325 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:16:43.802331 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:16:43.802552 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:16:43.802751 (+ 199us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:16:43.803623 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:43.804922 (+ 1299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:43.807672 (+ 2750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:43.809058 (+ 1386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:43.809191 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:43.809385 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0620 20:16:43.809923 (+ 538us) server_negotiation.cc:299] Negotiation successful
0620 20:16:43.810082 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3219,"thread_start_us":3115,"threads_started":1}
I20260620 20:16:43.811728 25138 tablet_service.cc:1467] Processing CreateTablet for tablet 252c1e5ecf4c4a4785a1f629c9a18638 (DEFAULT_TABLE table=test-table [id=9cf54558941f40b1b9bc8ccf61d79c6f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:16:43.813220 25138 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 252c1e5ecf4c4a4785a1f629c9a18638. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:16:43.812997 24998 tablet_service.cc:1467] Processing CreateTablet for tablet 252c1e5ecf4c4a4785a1f629c9a18638 (DEFAULT_TABLE table=test-table [id=9cf54558941f40b1b9bc8ccf61d79c6f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:16:43.813683 25278 tablet_service.cc:1467] Processing CreateTablet for tablet 252c1e5ecf4c4a4785a1f629c9a18638 (DEFAULT_TABLE table=test-table [id=9cf54558941f40b1b9bc8ccf61d79c6f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:16:43.814282 24998 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 252c1e5ecf4c4a4785a1f629c9a18638. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:16:43.814963 25278 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 252c1e5ecf4c4a4785a1f629c9a18638. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:16:43.826673 25373 tablet_bootstrap.cc:492] T 252c1e5ecf4c4a4785a1f629c9a18638 P 5a07c01f0567489fb51b6bb676a3764f: Bootstrap starting.
I20260620 20:16:43.829044 25375 tablet_bootstrap.cc:492] T 252c1e5ecf4c4a4785a1f629c9a18638 P 386309b3999d45769c3723f68adf1210: Bootstrap starting.
I20260620 20:16:43.830540 25373 tablet_bootstrap.cc:654] T 252c1e5ecf4c4a4785a1f629c9a18638 P 5a07c01f0567489fb51b6bb676a3764f: Neither blocks nor log segments found. Creating new log.
I20260620 20:16:43.831812 25373 log.cc:824] T 252c1e5ecf4c4a4785a1f629c9a18638 P 5a07c01f0567489fb51b6bb676a3764f: Log is configured to *not* fsync() on all Append() calls
I20260620 20:16:43.832602 25375 tablet_bootstrap.cc:654] T 252c1e5ecf4c4a4785a1f629c9a18638 P 386309b3999d45769c3723f68adf1210: Neither blocks nor log segments found. Creating new log.
I20260620 20:16:43.832643 25374 tablet_bootstrap.cc:492] T 252c1e5ecf4c4a4785a1f629c9a18638 P a64d4977929340b7a5de6108995a78fa: Bootstrap starting.
I20260620 20:16:43.834354 25375 log.cc:824] T 252c1e5ecf4c4a4785a1f629c9a18638 P 386309b3999d45769c3723f68adf1210: Log is configured to *not* fsync() on all Append() calls
I20260620 20:16:43.835875 25374 tablet_bootstrap.cc:654] T 252c1e5ecf4c4a4785a1f629c9a18638 P a64d4977929340b7a5de6108995a78fa: Neither blocks nor log segments found. Creating new log.
I20260620 20:16:43.836807 25374 log.cc:824] T 252c1e5ecf4c4a4785a1f629c9a18638 P a64d4977929340b7a5de6108995a78fa: Log is configured to *not* fsync() on all Append() calls
I20260620 20:16:43.839561 25374 tablet_bootstrap.cc:492] T 252c1e5ecf4c4a4785a1f629c9a18638 P a64d4977929340b7a5de6108995a78fa: No bootstrap required, opened a new log
I20260620 20:16:43.839845 25374 ts_tablet_manager.cc:1397] T 252c1e5ecf4c4a4785a1f629c9a18638 P a64d4977929340b7a5de6108995a78fa: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260620 20:16:43.839915 25375 tablet_bootstrap.cc:492] T 252c1e5ecf4c4a4785a1f629c9a18638 P 386309b3999d45769c3723f68adf1210: No bootstrap required, opened a new log
I20260620 20:16:43.839949 25373 tablet_bootstrap.cc:492] T 252c1e5ecf4c4a4785a1f629c9a18638 P 5a07c01f0567489fb51b6bb676a3764f: No bootstrap required, opened a new log
I20260620 20:16:43.840183 25373 ts_tablet_manager.cc:1397] T 252c1e5ecf4c4a4785a1f629c9a18638 P 5a07c01f0567489fb51b6bb676a3764f: Time spent bootstrapping tablet: real 0.014s user 0.011s sys 0.000s
I20260620 20:16:43.840183 25375 ts_tablet_manager.cc:1397] T 252c1e5ecf4c4a4785a1f629c9a18638 P 386309b3999d45769c3723f68adf1210: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.005s
I20260620 20:16:43.847879 25374 raft_consensus.cc:348] T 252c1e5ecf4c4a4785a1f629c9a18638 P a64d4977929340b7a5de6108995a78fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64d4977929340b7a5de6108995a78fa" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 36825 } } peers { permanent_uuid: "5a07c01f0567489fb51b6bb676a3764f" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 43535 } } peers { permanent_uuid: "386309b3999d45769c3723f68adf1210" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 44465 } }
I20260620 20:16:43.848254 25374 raft_consensus.cc:374] T 252c1e5ecf4c4a4785a1f629c9a18638 P a64d4977929340b7a5de6108995a78fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:16:43.848361 25374 raft_consensus.cc:729] T 252c1e5ecf4c4a4785a1f629c9a18638 P a64d4977929340b7a5de6108995a78fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64d4977929340b7a5de6108995a78fa, State: Initialized, Role: FOLLOWER
I20260620 20:16:43.848969 25374 consensus_queue.cc:260] T 252c1e5ecf4c4a4785a1f629c9a18638 P a64d4977929340b7a5de6108995a78fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64d4977929340b7a5de6108995a78fa" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 36825 } } peers { permanent_uuid: "5a07c01f0567489fb51b6bb676a3764f" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 43535 } } peers { permanent_uuid: "386309b3999d45769c3723f68adf1210" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 44465 } }
I20260620 20:16:43.850791 25374 ts_tablet_manager.cc:1428] T 252c1e5ecf4c4a4785a1f629c9a18638 P a64d4977929340b7a5de6108995a78fa: Time spent starting tablet: real 0.011s user 0.005s sys 0.006s
I20260620 20:16:43.850749 25375 raft_consensus.cc:348] T 252c1e5ecf4c4a4785a1f629c9a18638 P 386309b3999d45769c3723f68adf1210 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64d4977929340b7a5de6108995a78fa" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 36825 } } peers { permanent_uuid: "5a07c01f0567489fb51b6bb676a3764f" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 43535 } } peers { permanent_uuid: "386309b3999d45769c3723f68adf1210" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 44465 } }
I20260620 20:16:43.850956 25373 raft_consensus.cc:348] T 252c1e5ecf4c4a4785a1f629c9a18638 P 5a07c01f0567489fb51b6bb676a3764f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64d4977929340b7a5de6108995a78fa" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 36825 } } peers { permanent_uuid: "5a07c01f0567489fb51b6bb676a3764f" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 43535 } } peers { permanent_uuid: "386309b3999d45769c3723f68adf1210" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 44465 } }
I20260620 20:16:43.851289 25375 raft_consensus.cc:374] T 252c1e5ecf4c4a4785a1f629c9a18638 P 386309b3999d45769c3723f68adf1210 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:16:43.851409 25373 raft_consensus.cc:374] T 252c1e5ecf4c4a4785a1f629c9a18638 P 5a07c01f0567489fb51b6bb676a3764f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:16:43.851428 25375 raft_consensus.cc:729] T 252c1e5ecf4c4a4785a1f629c9a18638 P 386309b3999d45769c3723f68adf1210 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 386309b3999d45769c3723f68adf1210, State: Initialized, Role: FOLLOWER
I20260620 20:16:43.851517 25373 raft_consensus.cc:729] T 252c1e5ecf4c4a4785a1f629c9a18638 P 5a07c01f0567489fb51b6bb676a3764f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a07c01f0567489fb51b6bb676a3764f, State: Initialized, Role: FOLLOWER
I20260620 20:16:43.852097 25373 consensus_queue.cc:260] T 252c1e5ecf4c4a4785a1f629c9a18638 P 5a07c01f0567489fb51b6bb676a3764f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64d4977929340b7a5de6108995a78fa" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 36825 } } peers { permanent_uuid: "5a07c01f0567489fb51b6bb676a3764f" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 43535 } } peers { permanent_uuid: "386309b3999d45769c3723f68adf1210" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 44465 } }
I20260620 20:16:43.852097 25375 consensus_queue.cc:260] T 252c1e5ecf4c4a4785a1f629c9a18638 P 386309b3999d45769c3723f68adf1210 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64d4977929340b7a5de6108995a78fa" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 36825 } } peers { permanent_uuid: "5a07c01f0567489fb51b6bb676a3764f" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 43535 } } peers { permanent_uuid: "386309b3999d45769c3723f68adf1210" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 44465 } }
I20260620 20:16:43.855822 25373 ts_tablet_manager.cc:1428] T 252c1e5ecf4c4a4785a1f629c9a18638 P 5a07c01f0567489fb51b6bb676a3764f: Time spent starting tablet: real 0.016s user 0.013s sys 0.003s
I20260620 20:16:43.856107 25375 ts_tablet_manager.cc:1428] T 252c1e5ecf4c4a4785a1f629c9a18638 P 386309b3999d45769c3723f68adf1210: Time spent starting tablet: real 0.016s user 0.012s sys 0.004s
I20260620 20:16:44.007227 25381 raft_consensus.cc:482] T 252c1e5ecf4c4a4785a1f629c9a18638 P 5a07c01f0567489fb51b6bb676a3764f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 20:16:44.007601 25381 raft_consensus.cc:504] T 252c1e5ecf4c4a4785a1f629c9a18638 P 5a07c01f0567489fb51b6bb676a3764f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64d4977929340b7a5de6108995a78fa" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 36825 } } peers { permanent_uuid: "5a07c01f0567489fb51b6bb676a3764f" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 43535 } } peers { permanent_uuid: "386309b3999d45769c3723f68adf1210" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 44465 } }
I20260620 20:16:44.009244 25381 leader_election.cc:288] T 252c1e5ecf4c4a4785a1f629c9a18638 P 5a07c01f0567489fb51b6bb676a3764f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a64d4977929340b7a5de6108995a78fa (127.16.217.2:36825), 386309b3999d45769c3723f68adf1210 (127.16.217.3:44465)
I20260620 20:16:44.016767 25379 raft_consensus.cc:482] T 252c1e5ecf4c4a4785a1f629c9a18638 P a64d4977929340b7a5de6108995a78fa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 20:16:44.017184 25379 raft_consensus.cc:504] T 252c1e5ecf4c4a4785a1f629c9a18638 P a64d4977929340b7a5de6108995a78fa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64d4977929340b7a5de6108995a78fa" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 36825 } } peers { permanent_uuid: "5a07c01f0567489fb51b6bb676a3764f" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 43535 } } peers { permanent_uuid: "386309b3999d45769c3723f68adf1210" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 44465 } }
I20260620 20:16:44.018460 25371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:44.011082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.217.1:53111 (local address 127.16.217.3:44465)
0620 20:16:44.011325 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0620 20:16:44.011329 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:16:44.011399 (+ 70us) server_negotiation.cc:407] Connection header received
0620 20:16:44.011578 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:16:44.011581 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:16:44.011638 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:16:44.011753 (+ 115us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:16:44.012239 (+ 486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:44.013283 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:44.015972 (+ 2689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:44.017254 (+ 1282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:44.017429 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:44.017510 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0620 20:16:44.017830 (+ 320us) server_negotiation.cc:299] Negotiation successful
0620 20:16:44.018223 (+ 393us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260620 20:16:44.018505 25369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:44.009930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.217.1:50525 (local address 127.16.217.2:36825)
0620 20:16:44.010124 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0620 20:16:44.010129 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:16:44.010661 (+ 532us) server_negotiation.cc:407] Connection header received
0620 20:16:44.010876 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:16:44.010885 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:16:44.010978 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:16:44.011093 (+ 115us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:16:44.011890 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:44.013019 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:44.015980 (+ 2961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:44.017361 (+ 1381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:44.017511 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:44.017586 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 20:16:44.018172 (+ 586us) server_negotiation.cc:299] Negotiation successful
0620 20:16:44.018294 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 20:16:44.018625 25383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:44.010658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.3:44465 (local address 127.16.217.1:53111)
0620 20:16:44.011303 (+ 645us) negotiation.cc:107] Waiting for socket to connect
0620 20:16:44.011318 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 20:16:44.011459 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:16:44.011794 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:16:44.011800 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:16:44.011828 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:16:44.012090 (+ 262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:16:44.012097 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:16:44.013408 (+ 1311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:16:44.013413 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:16:44.015794 (+ 2381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:16:44.015808 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:16:44.017469 (+ 1661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:16:44.017475 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:16:44.017635 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:44.017681 (+ 46us) client_negotiation.cc:769] Sending connection context
0620 20:16:44.017804 (+ 123us) client_negotiation.cc:240] Negotiation successful
0620 20:16:44.018266 (+ 462us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":513,"mutex_wait_us":70,"thread_start_us":349,"threads_started":1}
I20260620 20:16:44.019176 25382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:44.009801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.2:36825 (local address 127.16.217.1:50525)
0620 20:16:44.010525 (+ 724us) negotiation.cc:107] Waiting for socket to connect
0620 20:16:44.010546 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 20:16:44.010695 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:16:44.011273 (+ 578us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:16:44.011279 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:16:44.011312 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:16:44.011714 (+ 402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:16:44.011747 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:16:44.013165 (+ 1418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:16:44.013168 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:16:44.015801 (+ 2633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:16:44.015809 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:16:44.017778 (+ 1969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:16:44.017785 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:16:44.017993 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:44.018026 (+ 33us) client_negotiation.cc:769] Sending connection context
0620 20:16:44.018726 (+ 700us) client_negotiation.cc:240] Negotiation successful
0620 20:16:44.018885 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":595,"mutex_wait_us":108,"thread_start_us":405,"threads_started":1}
I20260620 20:16:44.019332 25379 leader_election.cc:288] T 252c1e5ecf4c4a4785a1f629c9a18638 P a64d4977929340b7a5de6108995a78fa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5a07c01f0567489fb51b6bb676a3764f (127.16.217.1:43535), 386309b3999d45769c3723f68adf1210 (127.16.217.3:44465)
I20260620 20:16:44.019892 25303 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "252c1e5ecf4c4a4785a1f629c9a18638" candidate_uuid: "5a07c01f0567489fb51b6bb676a3764f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "386309b3999d45769c3723f68adf1210" is_pre_election: true
I20260620 20:16:44.020687 25303 raft_consensus.cc:2454] T 252c1e5ecf4c4a4785a1f629c9a18638 P 386309b3999d45769c3723f68adf1210 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5a07c01f0567489fb51b6bb676a3764f in term 0.
I20260620 20:16:44.021551 24948 leader_election.cc:302] T 252c1e5ecf4c4a4785a1f629c9a18638 P 5a07c01f0567489fb51b6bb676a3764f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 386309b3999d45769c3723f68adf1210, 5a07c01f0567489fb51b6bb676a3764f; no voters:
I20260620 20:16:44.021589 25158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "252c1e5ecf4c4a4785a1f629c9a18638" candidate_uuid: "5a07c01f0567489fb51b6bb676a3764f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64d4977929340b7a5de6108995a78fa" is_pre_election: true
I20260620 20:16:44.022259 25381 raft_consensus.cc:2781] T 252c1e5ecf4c4a4785a1f629c9a18638 P 5a07c01f0567489fb51b6bb676a3764f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 20:16:44.022202 25158 raft_consensus.cc:2454] T 252c1e5ecf4c4a4785a1f629c9a18638 P a64d4977929340b7a5de6108995a78fa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5a07c01f0567489fb51b6bb676a3764f in term 0.
I20260620 20:16:44.022475 25381 raft_consensus.cc:482] T 252c1e5ecf4c4a4785a1f629c9a18638 P 5a07c01f0567489fb51b6bb676a3764f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 20:16:44.022588 25381 raft_consensus.cc:3037] T 252c1e5ecf4c4a4785a1f629c9a18638 P 5a07c01f0567489fb51b6bb676a3764f [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:16:44.025964 25381 raft_consensus.cc:504] T 252c1e5ecf4c4a4785a1f629c9a18638 P 5a07c01f0567489fb51b6bb676a3764f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64d4977929340b7a5de6108995a78fa" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 36825 } } peers { permanent_uuid: "5a07c01f0567489fb51b6bb676a3764f" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 43535 } } peers { permanent_uuid: "386309b3999d45769c3723f68adf1210" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 44465 } }
I20260620 20:16:44.026611 25381 leader_election.cc:288] T 252c1e5ecf4c4a4785a1f629c9a18638 P 5a07c01f0567489fb51b6bb676a3764f [CANDIDATE]: Term 1 election: Requested vote from peers a64d4977929340b7a5de6108995a78fa (127.16.217.2:36825), 386309b3999d45769c3723f68adf1210 (127.16.217.3:44465)
I20260620 20:16:44.027259 25158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "252c1e5ecf4c4a4785a1f629c9a18638" candidate_uuid: "5a07c01f0567489fb51b6bb676a3764f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64d4977929340b7a5de6108995a78fa"
I20260620 20:16:44.027473 25158 raft_consensus.cc:3037] T 252c1e5ecf4c4a4785a1f629c9a18638 P a64d4977929340b7a5de6108995a78fa [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:16:44.030457 25370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:44.019903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.217.2:42309 (local address 127.16.217.1:43535)
0620 20:16:44.020089 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0620 20:16:44.020096 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:16:44.024093 (+ 3997us) server_negotiation.cc:407] Connection header received
0620 20:16:44.024341 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:16:44.024345 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:16:44.024419 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:16:44.024517 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:16:44.025168 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:44.026305 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:44.028476 (+ 2171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:44.029315 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:44.029480 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:44.029554 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 20:16:44.029849 (+ 295us) server_negotiation.cc:299] Negotiation successful
0620 20:16:44.030256 (+ 407us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 20:16:44.030961 25303 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "252c1e5ecf4c4a4785a1f629c9a18638" candidate_uuid: "5a07c01f0567489fb51b6bb676a3764f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "386309b3999d45769c3723f68adf1210"
I20260620 20:16:44.031193 25385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:44.019481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.1:43535 (local address 127.16.217.2:42309)
0620 20:16:44.023997 (+ 4516us) negotiation.cc:107] Waiting for socket to connect
0620 20:16:44.024018 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 20:16:44.024185 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:16:44.024669 (+ 484us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:16:44.024673 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:16:44.024695 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:16:44.025035 (+ 340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:16:44.025043 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:16:44.026458 (+ 1415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:16:44.026465 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:16:44.028331 (+ 1866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:16:44.028343 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:16:44.029530 (+ 1187us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:16:44.029536 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:16:44.029689 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:44.029712 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 20:16:44.029816 (+ 104us) client_negotiation.cc:240] Negotiation successful
0620 20:16:44.029961 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4324,"mutex_wait_us":71,"thread_start_us":4442,"threads_started":1}
I20260620 20:16:44.031164 25303 raft_consensus.cc:3037] T 252c1e5ecf4c4a4785a1f629c9a18638 P 386309b3999d45769c3723f68adf1210 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:16:44.031711 25371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:44.020004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.217.2:51239 (local address 127.16.217.3:44465)
0620 20:16:44.020909 (+ 905us) server_negotiation.cc:206] Beginning negotiation
0620 20:16:44.020915 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:16:44.024291 (+ 3376us) server_negotiation.cc:407] Connection header received
0620 20:16:44.024515 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:16:44.024521 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:16:44.024585 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:16:44.024668 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:16:44.025365 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:44.026438 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:44.029105 (+ 2667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:44.030501 (+ 1396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:44.030624 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:44.030688 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0620 20:16:44.031292 (+ 604us) server_negotiation.cc:299] Negotiation successful
0620 20:16:44.031428 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":793}
I20260620 20:16:44.032003 25018 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "252c1e5ecf4c4a4785a1f629c9a18638" candidate_uuid: "a64d4977929340b7a5de6108995a78fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a07c01f0567489fb51b6bb676a3764f" is_pre_election: true
I20260620 20:16:44.032341 25018 raft_consensus.cc:2379] T 252c1e5ecf4c4a4785a1f629c9a18638 P 5a07c01f0567489fb51b6bb676a3764f [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a64d4977929340b7a5de6108995a78fa in current term 1: Already voted for candidate 5a07c01f0567489fb51b6bb676a3764f in this term.
I20260620 20:16:44.032457 25384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:44.019851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.3:44465 (local address 127.16.217.2:51239)
0620 20:16:44.024190 (+ 4339us) negotiation.cc:107] Waiting for socket to connect
0620 20:16:44.024212 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 20:16:44.024381 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:16:44.024697 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:16:44.024701 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:16:44.024725 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:16:44.025192 (+ 467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:16:44.025205 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:16:44.026585 (+ 1380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:16:44.026592 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:16:44.028929 (+ 2337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:16:44.028942 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:16:44.030877 (+ 1935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:16:44.030883 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:16:44.031064 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:44.031100 (+ 36us) client_negotiation.cc:769] Sending connection context
0620 20:16:44.031945 (+ 845us) client_negotiation.cc:240] Negotiation successful
0620 20:16:44.032079 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4224,"mutex_wait_us":21,"thread_start_us":3715,"threads_started":1}
I20260620 20:16:44.033174 25302 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "252c1e5ecf4c4a4785a1f629c9a18638" candidate_uuid: "a64d4977929340b7a5de6108995a78fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "386309b3999d45769c3723f68adf1210" is_pre_election: true
I20260620 20:16:44.034109 25089 leader_election.cc:302] T 252c1e5ecf4c4a4785a1f629c9a18638 P a64d4977929340b7a5de6108995a78fa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a64d4977929340b7a5de6108995a78fa; no voters: 386309b3999d45769c3723f68adf1210, 5a07c01f0567489fb51b6bb676a3764f
I20260620 20:16:44.034482 25158 raft_consensus.cc:2454] T 252c1e5ecf4c4a4785a1f629c9a18638 P a64d4977929340b7a5de6108995a78fa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a07c01f0567489fb51b6bb676a3764f in term 1.
I20260620 20:16:44.034531 25303 raft_consensus.cc:2454] T 252c1e5ecf4c4a4785a1f629c9a18638 P 386309b3999d45769c3723f68adf1210 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a07c01f0567489fb51b6bb676a3764f in term 1.
I20260620 20:16:44.034668 25379 raft_consensus.cc:2726] T 252c1e5ecf4c4a4785a1f629c9a18638 P a64d4977929340b7a5de6108995a78fa [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260620 20:16:44.034873 24948 leader_election.cc:302] T 252c1e5ecf4c4a4785a1f629c9a18638 P 5a07c01f0567489fb51b6bb676a3764f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a07c01f0567489fb51b6bb676a3764f, a64d4977929340b7a5de6108995a78fa; no voters:
I20260620 20:16:44.035132 25381 raft_consensus.cc:2781] T 252c1e5ecf4c4a4785a1f629c9a18638 P 5a07c01f0567489fb51b6bb676a3764f [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:16:44.035578 25381 raft_consensus.cc:686] T 252c1e5ecf4c4a4785a1f629c9a18638 P 5a07c01f0567489fb51b6bb676a3764f [term 1 LEADER]: Becoming Leader. State: Replica: 5a07c01f0567489fb51b6bb676a3764f, State: Running, Role: LEADER
I20260620 20:16:44.036038 25381 consensus_queue.cc:237] T 252c1e5ecf4c4a4785a1f629c9a18638 P 5a07c01f0567489fb51b6bb676a3764f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64d4977929340b7a5de6108995a78fa" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 36825 } } peers { permanent_uuid: "5a07c01f0567489fb51b6bb676a3764f" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 43535 } } peers { permanent_uuid: "386309b3999d45769c3723f68adf1210" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 44465 } }
I20260620 20:16:44.040458 24860 catalog_manager.cc:5495] T 252c1e5ecf4c4a4785a1f629c9a18638 P 5a07c01f0567489fb51b6bb676a3764f reported cstate change: term changed from 0 to 1, leader changed from <none> to 5a07c01f0567489fb51b6bb676a3764f (127.16.217.1). New cstate: current_term: 1 leader_uuid: "5a07c01f0567489fb51b6bb676a3764f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64d4977929340b7a5de6108995a78fa" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 36825 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5a07c01f0567489fb51b6bb676a3764f" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 43535 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "386309b3999d45769c3723f68adf1210" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 44465 } health_report { overall_health: UNKNOWN } } }
W20260620 20:16:44.044088 25069 tablet.cc:2369] T 252c1e5ecf4c4a4785a1f629c9a18638 P 5a07c01f0567489fb51b6bb676a3764f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 20:16:44.058553 25209 tablet.cc:2369] T 252c1e5ecf4c4a4785a1f629c9a18638 P a64d4977929340b7a5de6108995a78fa: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 20:16:44.085180 25349 tablet.cc:2369] T 252c1e5ecf4c4a4785a1f629c9a18638 P 386309b3999d45769c3723f68adf1210: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 20:16:44.099965 25370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:44.094580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38012 (local address 127.16.217.1:43535)
0620 20:16:44.094743 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0620 20:16:44.094748 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:16:44.094801 (+ 53us) server_negotiation.cc:407] Connection header received
0620 20:16:44.095032 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:16:44.095036 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:16:44.095122 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:16:44.095253 (+ 131us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 20:16:44.095602 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:44.096342 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:44.097752 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:44.098197 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:44.098287 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:44.098791 (+ 504us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 20:16:44.099315 (+ 524us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 20:16:44.099421 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0620 20:16:44.099645 (+ 224us) server_negotiation.cc:299] Negotiation successful
0620 20:16:44.099709 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 20:16:44.107872 25158 raft_consensus.cc:1261] T 252c1e5ecf4c4a4785a1f629c9a18638 P a64d4977929340b7a5de6108995a78fa [term 1 FOLLOWER]: Refusing update from remote peer 5a07c01f0567489fb51b6bb676a3764f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:16:44.107888 25303 raft_consensus.cc:1261] T 252c1e5ecf4c4a4785a1f629c9a18638 P 386309b3999d45769c3723f68adf1210 [term 1 FOLLOWER]: Refusing update from remote peer 5a07c01f0567489fb51b6bb676a3764f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:16:44.108702 25386 consensus_queue.cc:1035] T 252c1e5ecf4c4a4785a1f629c9a18638 P 5a07c01f0567489fb51b6bb676a3764f [LEADER]: Connected to new peer: Peer: permanent_uuid: "386309b3999d45769c3723f68adf1210" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 44465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:16:44.109030 25381 consensus_queue.cc:1035] T 252c1e5ecf4c4a4785a1f629c9a18638 P 5a07c01f0567489fb51b6bb676a3764f [LEADER]: Connected to new peer: Peer: permanent_uuid: "a64d4977929340b7a5de6108995a78fa" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 36825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:16:44.119800 25389 mvcc.cc:205] Tried to move back new op lower bound from 7299017130416234496 to 7299017130135056384. Current Snapshot: MvccSnapshot[applied={T|T < 7299017130416234496}]
I20260620 20:16:44.133991 24859 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40564:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 20:16:44.134276 24859 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40564:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 20:16:44.137856 24859 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 827ffa1470674fdfa9e771771e6a7d94: Sending DeleteTablet for 3 replicas of tablet 252c1e5ecf4c4a4785a1f629c9a18638
I20260620 20:16:44.139022 25138 tablet_service.cc:1514] Processing DeleteTablet for tablet 252c1e5ecf4c4a4785a1f629c9a18638 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:16:44 UTC) from {username='oryx', principal='oryx/127.16.217.62@KRBTEST.COM'} at 127.0.0.1:38734
I20260620 20:16:44.139078 24998 tablet_service.cc:1514] Processing DeleteTablet for tablet 252c1e5ecf4c4a4785a1f629c9a18638 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:16:44 UTC) from {username='oryx', principal='oryx/127.16.217.62@KRBTEST.COM'} at 127.0.0.1:38004
I20260620 20:16:44.139170 25278 tablet_service.cc:1514] Processing DeleteTablet for tablet 252c1e5ecf4c4a4785a1f629c9a18638 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:16:44 UTC) from {username='oryx', principal='oryx/127.16.217.62@KRBTEST.COM'} at 127.0.0.1:49776
I20260620 20:16:44.142030 24859 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40564:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 20:16:44.143343 24859 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 20:16:44.150486 25399 tablet_replica.cc:330] T 252c1e5ecf4c4a4785a1f629c9a18638 P a64d4977929340b7a5de6108995a78fa: stopping tablet replica
I20260620 20:16:44.151248 25399 raft_consensus.cc:2229] T 252c1e5ecf4c4a4785a1f629c9a18638 P a64d4977929340b7a5de6108995a78fa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 20:16:44.153220 25399 raft_consensus.cc:2258] T 252c1e5ecf4c4a4785a1f629c9a18638 P a64d4977929340b7a5de6108995a78fa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:16:44.153187 24997 tablet_service.cc:1467] Processing CreateTablet for tablet 28eab8dc853f47ffac0efd6edc989abe (DEFAULT_TABLE table=test-table [id=d54f0c3ee18643eaaa5581dbbd6cad98]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:16:44.153824 24997 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 28eab8dc853f47ffac0efd6edc989abe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:16:44.154054 25138 tablet_service.cc:1467] Processing CreateTablet for tablet 28eab8dc853f47ffac0efd6edc989abe (DEFAULT_TABLE table=test-table [id=d54f0c3ee18643eaaa5581dbbd6cad98]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:16:44.154289 25401 tablet_replica.cc:330] T 252c1e5ecf4c4a4785a1f629c9a18638 P 386309b3999d45769c3723f68adf1210: stopping tablet replica
I20260620 20:16:44.154455 25138 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 28eab8dc853f47ffac0efd6edc989abe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:16:44.154717 25401 raft_consensus.cc:2229] T 252c1e5ecf4c4a4785a1f629c9a18638 P 386309b3999d45769c3723f68adf1210 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 20:16:44.155138 25400 tablet_replica.cc:330] T 252c1e5ecf4c4a4785a1f629c9a18638 P 5a07c01f0567489fb51b6bb676a3764f: stopping tablet replica
I20260620 20:16:44.155565 25400 raft_consensus.cc:2229] T 252c1e5ecf4c4a4785a1f629c9a18638 P 5a07c01f0567489fb51b6bb676a3764f [term 1 LEADER]: Raft consensus shutting down.
I20260620 20:16:44.156193 25400 raft_consensus.cc:2258] T 252c1e5ecf4c4a4785a1f629c9a18638 P 5a07c01f0567489fb51b6bb676a3764f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:16:44.156987 25399 ts_tablet_manager.cc:1905] T 252c1e5ecf4c4a4785a1f629c9a18638 P a64d4977929340b7a5de6108995a78fa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:16:44.158946 25277 tablet_service.cc:1467] Processing CreateTablet for tablet 28eab8dc853f47ffac0efd6edc989abe (DEFAULT_TABLE table=test-table [id=d54f0c3ee18643eaaa5581dbbd6cad98]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:16:44.159436 25277 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 28eab8dc853f47ffac0efd6edc989abe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:16:44.160707 25374 tablet_bootstrap.cc:492] T 28eab8dc853f47ffac0efd6edc989abe P a64d4977929340b7a5de6108995a78fa: Bootstrap starting.
I20260620 20:16:44.162034 25400 ts_tablet_manager.cc:1905] T 252c1e5ecf4c4a4785a1f629c9a18638 P 5a07c01f0567489fb51b6bb676a3764f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:16:44.162078 25401 raft_consensus.cc:2258] T 252c1e5ecf4c4a4785a1f629c9a18638 P 386309b3999d45769c3723f68adf1210 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:16:44.162324 25374 tablet_bootstrap.cc:654] T 28eab8dc853f47ffac0efd6edc989abe P a64d4977929340b7a5de6108995a78fa: Neither blocks nor log segments found. Creating new log.
I20260620 20:16:44.165091 25375 tablet_bootstrap.cc:492] T 28eab8dc853f47ffac0efd6edc989abe P 386309b3999d45769c3723f68adf1210: Bootstrap starting.
I20260620 20:16:44.165401 25401 ts_tablet_manager.cc:1905] T 252c1e5ecf4c4a4785a1f629c9a18638 P 386309b3999d45769c3723f68adf1210: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:16:44.165946 25373 tablet_bootstrap.cc:492] T 28eab8dc853f47ffac0efd6edc989abe P 5a07c01f0567489fb51b6bb676a3764f: Bootstrap starting.
I20260620 20:16:44.166673 25375 tablet_bootstrap.cc:654] T 28eab8dc853f47ffac0efd6edc989abe P 386309b3999d45769c3723f68adf1210: Neither blocks nor log segments found. Creating new log.
I20260620 20:16:44.167548 25373 tablet_bootstrap.cc:654] T 28eab8dc853f47ffac0efd6edc989abe P 5a07c01f0567489fb51b6bb676a3764f: Neither blocks nor log segments found. Creating new log.
I20260620 20:16:44.167709 25399 ts_tablet_manager.cc:1918] T 252c1e5ecf4c4a4785a1f629c9a18638 P a64d4977929340b7a5de6108995a78fa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 20:16:44.167917 25399 log.cc:1196] T 252c1e5ecf4c4a4785a1f629c9a18638 P a64d4977929340b7a5de6108995a78fa: Deleting WAL directory at /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986536687152-17252-0/minicluster-data/ts-1/wal/wals/252c1e5ecf4c4a4785a1f629c9a18638
I20260620 20:16:44.168565 25399 ts_tablet_manager.cc:1939] T 252c1e5ecf4c4a4785a1f629c9a18638 P a64d4977929340b7a5de6108995a78fa: Deleting consensus metadata
I20260620 20:16:44.170543 24843 catalog_manager.cc:4841] TS a64d4977929340b7a5de6108995a78fa (127.16.217.2:36825): tablet 252c1e5ecf4c4a4785a1f629c9a18638 (table test-table [id=9cf54558941f40b1b9bc8ccf61d79c6f]) successfully deleted
I20260620 20:16:44.178318 25400 ts_tablet_manager.cc:1918] T 252c1e5ecf4c4a4785a1f629c9a18638 P 5a07c01f0567489fb51b6bb676a3764f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 20:16:44.184731 25374 tablet_bootstrap.cc:492] T 28eab8dc853f47ffac0efd6edc989abe P a64d4977929340b7a5de6108995a78fa: No bootstrap required, opened a new log
I20260620 20:16:44.184990 25374 ts_tablet_manager.cc:1397] T 28eab8dc853f47ffac0efd6edc989abe P a64d4977929340b7a5de6108995a78fa: Time spent bootstrapping tablet: real 0.024s user 0.009s sys 0.003s
I20260620 20:16:44.185949 25374 raft_consensus.cc:348] T 28eab8dc853f47ffac0efd6edc989abe P a64d4977929340b7a5de6108995a78fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a07c01f0567489fb51b6bb676a3764f" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 43535 } } peers { permanent_uuid: "a64d4977929340b7a5de6108995a78fa" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 36825 } } peers { permanent_uuid: "386309b3999d45769c3723f68adf1210" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 44465 } }
I20260620 20:16:44.187335 25373 tablet_bootstrap.cc:492] T 28eab8dc853f47ffac0efd6edc989abe P 5a07c01f0567489fb51b6bb676a3764f: No bootstrap required, opened a new log
I20260620 20:16:44.187407 25401 ts_tablet_manager.cc:1918] T 252c1e5ecf4c4a4785a1f629c9a18638 P 386309b3999d45769c3723f68adf1210: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 20:16:44.187618 25373 ts_tablet_manager.cc:1397] T 28eab8dc853f47ffac0efd6edc989abe P 5a07c01f0567489fb51b6bb676a3764f: Time spent bootstrapping tablet: real 0.022s user 0.001s sys 0.009s
I20260620 20:16:44.187635 25401 log.cc:1196] T 252c1e5ecf4c4a4785a1f629c9a18638 P 386309b3999d45769c3723f68adf1210: Deleting WAL directory at /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986536687152-17252-0/minicluster-data/ts-2/wal/wals/252c1e5ecf4c4a4785a1f629c9a18638
I20260620 20:16:44.188295 25401 ts_tablet_manager.cc:1939] T 252c1e5ecf4c4a4785a1f629c9a18638 P 386309b3999d45769c3723f68adf1210: Deleting consensus metadata
I20260620 20:16:44.188581 25373 raft_consensus.cc:348] T 28eab8dc853f47ffac0efd6edc989abe P 5a07c01f0567489fb51b6bb676a3764f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a07c01f0567489fb51b6bb676a3764f" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 43535 } } peers { permanent_uuid: "a64d4977929340b7a5de6108995a78fa" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 36825 } } peers { permanent_uuid: "386309b3999d45769c3723f68adf1210" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 44465 } }
I20260620 20:16:44.188969 25373 raft_consensus.cc:374] T 28eab8dc853f47ffac0efd6edc989abe P 5a07c01f0567489fb51b6bb676a3764f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:16:44.189113 25373 raft_consensus.cc:729] T 28eab8dc853f47ffac0efd6edc989abe P 5a07c01f0567489fb51b6bb676a3764f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a07c01f0567489fb51b6bb676a3764f, State: Initialized, Role: FOLLOWER
I20260620 20:16:44.189463 25373 consensus_queue.cc:260] T 28eab8dc853f47ffac0efd6edc989abe P 5a07c01f0567489fb51b6bb676a3764f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a07c01f0567489fb51b6bb676a3764f" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 43535 } } peers { permanent_uuid: "a64d4977929340b7a5de6108995a78fa" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 36825 } } peers { permanent_uuid: "386309b3999d45769c3723f68adf1210" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 44465 } }
I20260620 20:16:44.189967 25400 log.cc:1196] T 252c1e5ecf4c4a4785a1f629c9a18638 P 5a07c01f0567489fb51b6bb676a3764f: Deleting WAL directory at /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986536687152-17252-0/minicluster-data/ts-0/wal/wals/252c1e5ecf4c4a4785a1f629c9a18638
I20260620 20:16:44.190661 25400 ts_tablet_manager.cc:1939] T 252c1e5ecf4c4a4785a1f629c9a18638 P 5a07c01f0567489fb51b6bb676a3764f: Deleting consensus metadata
I20260620 20:16:44.192612 24846 catalog_manager.cc:4841] TS 5a07c01f0567489fb51b6bb676a3764f (127.16.217.1:43535): tablet 252c1e5ecf4c4a4785a1f629c9a18638 (table test-table [id=9cf54558941f40b1b9bc8ccf61d79c6f]) successfully deleted
I20260620 20:16:44.193004 25373 ts_tablet_manager.cc:1428] T 28eab8dc853f47ffac0efd6edc989abe P 5a07c01f0567489fb51b6bb676a3764f: Time spent starting tablet: real 0.005s user 0.000s sys 0.001s
I20260620 20:16:44.193011 24843 catalog_manager.cc:4841] TS 386309b3999d45769c3723f68adf1210 (127.16.217.3:44465): tablet 252c1e5ecf4c4a4785a1f629c9a18638 (table test-table [id=9cf54558941f40b1b9bc8ccf61d79c6f]) successfully deleted
I20260620 20:16:44.193580 25374 raft_consensus.cc:374] T 28eab8dc853f47ffac0efd6edc989abe P a64d4977929340b7a5de6108995a78fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:16:44.193754 25374 raft_consensus.cc:729] T 28eab8dc853f47ffac0efd6edc989abe P a64d4977929340b7a5de6108995a78fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64d4977929340b7a5de6108995a78fa, State: Initialized, Role: FOLLOWER
I20260620 20:16:44.194648 25375 tablet_bootstrap.cc:492] T 28eab8dc853f47ffac0efd6edc989abe P 386309b3999d45769c3723f68adf1210: No bootstrap required, opened a new log
I20260620 20:16:44.194850 25375 ts_tablet_manager.cc:1397] T 28eab8dc853f47ffac0efd6edc989abe P 386309b3999d45769c3723f68adf1210: Time spent bootstrapping tablet: real 0.030s user 0.007s sys 0.020s
I20260620 20:16:44.194725 25374 consensus_queue.cc:260] T 28eab8dc853f47ffac0efd6edc989abe P a64d4977929340b7a5de6108995a78fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a07c01f0567489fb51b6bb676a3764f" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 43535 } } peers { permanent_uuid: "a64d4977929340b7a5de6108995a78fa" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 36825 } } peers { permanent_uuid: "386309b3999d45769c3723f68adf1210" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 44465 } }
I20260620 20:16:44.195438 25374 ts_tablet_manager.cc:1428] T 28eab8dc853f47ffac0efd6edc989abe P a64d4977929340b7a5de6108995a78fa: Time spent starting tablet: real 0.010s user 0.002s sys 0.000s
I20260620 20:16:44.195669 25375 raft_consensus.cc:348] T 28eab8dc853f47ffac0efd6edc989abe P 386309b3999d45769c3723f68adf1210 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a07c01f0567489fb51b6bb676a3764f" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 43535 } } peers { permanent_uuid: "a64d4977929340b7a5de6108995a78fa" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 36825 } } peers { permanent_uuid: "386309b3999d45769c3723f68adf1210" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 44465 } }
I20260620 20:16:44.195909 25375 raft_consensus.cc:374] T 28eab8dc853f47ffac0efd6edc989abe P 386309b3999d45769c3723f68adf1210 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:16:44.196038 25375 raft_consensus.cc:729] T 28eab8dc853f47ffac0efd6edc989abe P 386309b3999d45769c3723f68adf1210 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 386309b3999d45769c3723f68adf1210, State: Initialized, Role: FOLLOWER
I20260620 20:16:44.196329 25375 consensus_queue.cc:260] T 28eab8dc853f47ffac0efd6edc989abe P 386309b3999d45769c3723f68adf1210 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a07c01f0567489fb51b6bb676a3764f" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 43535 } } peers { permanent_uuid: "a64d4977929340b7a5de6108995a78fa" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 36825 } } peers { permanent_uuid: "386309b3999d45769c3723f68adf1210" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 44465 } }
I20260620 20:16:44.196991 25375 ts_tablet_manager.cc:1428] T 28eab8dc853f47ffac0efd6edc989abe P 386309b3999d45769c3723f68adf1210: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260620 20:16:44.282552 25394 raft_consensus.cc:482] T 28eab8dc853f47ffac0efd6edc989abe P 5a07c01f0567489fb51b6bb676a3764f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 20:16:44.282835 25394 raft_consensus.cc:504] T 28eab8dc853f47ffac0efd6edc989abe P 5a07c01f0567489fb51b6bb676a3764f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a07c01f0567489fb51b6bb676a3764f" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 43535 } } peers { permanent_uuid: "a64d4977929340b7a5de6108995a78fa" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 36825 } } peers { permanent_uuid: "386309b3999d45769c3723f68adf1210" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 44465 } }
I20260620 20:16:44.283651 25394 leader_election.cc:288] T 28eab8dc853f47ffac0efd6edc989abe P 5a07c01f0567489fb51b6bb676a3764f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a64d4977929340b7a5de6108995a78fa (127.16.217.2:36825), 386309b3999d45769c3723f68adf1210 (127.16.217.3:44465)
I20260620 20:16:44.284276 25158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28eab8dc853f47ffac0efd6edc989abe" candidate_uuid: "5a07c01f0567489fb51b6bb676a3764f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64d4977929340b7a5de6108995a78fa" is_pre_election: true
I20260620 20:16:44.284322 25303 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28eab8dc853f47ffac0efd6edc989abe" candidate_uuid: "5a07c01f0567489fb51b6bb676a3764f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "386309b3999d45769c3723f68adf1210" is_pre_election: true
I20260620 20:16:44.284541 25158 raft_consensus.cc:2454] T 28eab8dc853f47ffac0efd6edc989abe P a64d4977929340b7a5de6108995a78fa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5a07c01f0567489fb51b6bb676a3764f in term 0.
I20260620 20:16:44.284626 25303 raft_consensus.cc:2454] T 28eab8dc853f47ffac0efd6edc989abe P 386309b3999d45769c3723f68adf1210 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5a07c01f0567489fb51b6bb676a3764f in term 0.
I20260620 20:16:44.285024 24948 leader_election.cc:302] T 28eab8dc853f47ffac0efd6edc989abe P 5a07c01f0567489fb51b6bb676a3764f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a07c01f0567489fb51b6bb676a3764f, a64d4977929340b7a5de6108995a78fa; no voters:
I20260620 20:16:44.285336 25394 raft_consensus.cc:2781] T 28eab8dc853f47ffac0efd6edc989abe P 5a07c01f0567489fb51b6bb676a3764f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 20:16:44.285472 25394 raft_consensus.cc:482] T 28eab8dc853f47ffac0efd6edc989abe P 5a07c01f0567489fb51b6bb676a3764f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 20:16:44.285598 25394 raft_consensus.cc:3037] T 28eab8dc853f47ffac0efd6edc989abe P 5a07c01f0567489fb51b6bb676a3764f [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:16:44.287892 25394 raft_consensus.cc:504] T 28eab8dc853f47ffac0efd6edc989abe P 5a07c01f0567489fb51b6bb676a3764f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a07c01f0567489fb51b6bb676a3764f" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 43535 } } peers { permanent_uuid: "a64d4977929340b7a5de6108995a78fa" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 36825 } } peers { permanent_uuid: "386309b3999d45769c3723f68adf1210" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 44465 } }
I20260620 20:16:44.288425 25394 leader_election.cc:288] T 28eab8dc853f47ffac0efd6edc989abe P 5a07c01f0567489fb51b6bb676a3764f [CANDIDATE]: Term 1 election: Requested vote from peers a64d4977929340b7a5de6108995a78fa (127.16.217.2:36825), 386309b3999d45769c3723f68adf1210 (127.16.217.3:44465)
I20260620 20:16:44.289003 25303 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28eab8dc853f47ffac0efd6edc989abe" candidate_uuid: "5a07c01f0567489fb51b6bb676a3764f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "386309b3999d45769c3723f68adf1210"
I20260620 20:16:44.289024 25158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28eab8dc853f47ffac0efd6edc989abe" candidate_uuid: "5a07c01f0567489fb51b6bb676a3764f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64d4977929340b7a5de6108995a78fa"
I20260620 20:16:44.289212 25303 raft_consensus.cc:3037] T 28eab8dc853f47ffac0efd6edc989abe P 386309b3999d45769c3723f68adf1210 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:16:44.289223 25158 raft_consensus.cc:3037] T 28eab8dc853f47ffac0efd6edc989abe P a64d4977929340b7a5de6108995a78fa [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:16:44.291281 25303 raft_consensus.cc:2454] T 28eab8dc853f47ffac0efd6edc989abe P 386309b3999d45769c3723f68adf1210 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a07c01f0567489fb51b6bb676a3764f in term 1.
I20260620 20:16:44.291692 24948 leader_election.cc:302] T 28eab8dc853f47ffac0efd6edc989abe P 5a07c01f0567489fb51b6bb676a3764f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 386309b3999d45769c3723f68adf1210, 5a07c01f0567489fb51b6bb676a3764f; no voters:
I20260620 20:16:44.291990 25394 raft_consensus.cc:2781] T 28eab8dc853f47ffac0efd6edc989abe P 5a07c01f0567489fb51b6bb676a3764f [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:16:44.292153 25394 raft_consensus.cc:686] T 28eab8dc853f47ffac0efd6edc989abe P 5a07c01f0567489fb51b6bb676a3764f [term 1 LEADER]: Becoming Leader. State: Replica: 5a07c01f0567489fb51b6bb676a3764f, State: Running, Role: LEADER
I20260620 20:16:44.292198 25158 raft_consensus.cc:2454] T 28eab8dc853f47ffac0efd6edc989abe P a64d4977929340b7a5de6108995a78fa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a07c01f0567489fb51b6bb676a3764f in term 1.
I20260620 20:16:44.292639 25394 consensus_queue.cc:237] T 28eab8dc853f47ffac0efd6edc989abe P 5a07c01f0567489fb51b6bb676a3764f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a07c01f0567489fb51b6bb676a3764f" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 43535 } } peers { permanent_uuid: "a64d4977929340b7a5de6108995a78fa" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 36825 } } peers { permanent_uuid: "386309b3999d45769c3723f68adf1210" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 44465 } }
I20260620 20:16:44.295509 24859 catalog_manager.cc:5495] T 28eab8dc853f47ffac0efd6edc989abe P 5a07c01f0567489fb51b6bb676a3764f reported cstate change: term changed from 0 to 1, leader changed from <none> to 5a07c01f0567489fb51b6bb676a3764f (127.16.217.1). New cstate: current_term: 1 leader_uuid: "5a07c01f0567489fb51b6bb676a3764f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a07c01f0567489fb51b6bb676a3764f" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 43535 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a64d4977929340b7a5de6108995a78fa" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 36825 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "386309b3999d45769c3723f68adf1210" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 44465 } health_report { overall_health: UNKNOWN } } }
Jun 20 20:16:44 dist-test-slave-2hpg krb5kdc[24814](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986600, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.217.62@KRBTEST.COM for oryx/127.16.217.62@KRBTEST.COM
I20260620 20:16:44.342303 25411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:44.328249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:39789 (local address 127.0.0.1:40566)
0620 20:16:44.328930 (+ 681us) negotiation.cc:107] Waiting for socket to connect
0620 20:16:44.328954 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 20:16:44.329119 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:16:44.329511 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:16:44.329520 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:16:44.330229 (+ 709us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:16:44.330410 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:16:44.330419 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:16:44.331506 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:16:44.331510 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:16:44.332474 (+ 964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:16:44.332492 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:16:44.333312 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:16:44.333317 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:16:44.333582 (+ 265us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:44.334439 (+ 857us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:16:44.334457 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:16:44.336743 (+ 2286us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:16:44.339788 (+ 3045us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:16:44.339799 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:16:44.339807 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:16:44.340189 (+ 382us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:16:44.340525 (+ 336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:16:44.340531 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:16:44.340534 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:16:44.340666 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:16:44.341158 (+ 492us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:16:44.341165 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:16:44.341322 (+ 157us) client_negotiation.cc:769] Sending connection context
0620 20:16:44.341530 (+ 208us) client_negotiation.cc:240] Negotiation successful
0620 20:16:44.341826 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":503,"thread_start_us":343,"threads_started":1}
I20260620 20:16:44.342675 25412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:44.328282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40566 (local address 127.16.217.62:39789)
0620 20:16:44.329232 (+ 950us) server_negotiation.cc:206] Beginning negotiation
0620 20:16:44.329239 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:16:44.329262 (+ 23us) server_negotiation.cc:407] Connection header received
0620 20:16:44.329300 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:16:44.329303 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:16:44.329353 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:16:44.329465 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:16:44.330548 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:44.331338 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:44.332624 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:44.333141 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:44.333265 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:44.336920 (+ 3655us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:16:44.336933 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:16:44.336936 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:16:44.336962 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:16:44.339652 (+ 2690us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:16:44.340299 (+ 647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:16:44.340307 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:16:44.340310 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:16:44.340423 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:16:44.340772 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:16:44.340779 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:16:44.340782 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:16:44.341028 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:16:44.341169 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0620 20:16:44.341640 (+ 471us) server_negotiation.cc:299] Negotiation successful
0620 20:16:44.342015 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":138,"server-negotiator.queue_time_us":819,"thread_start_us":663,"threads_started":1}
I20260620 20:16:44.347838 24859 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.16.217.62@KRBTEST.COM'} at 127.0.0.1:40566:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260620 20:16:44.349085 24859 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 20:16:44.355948 24997 tablet_service.cc:1467] Processing CreateTablet for tablet 2a177fdd3e9c489a9bafbfce528b6edd (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=d354fd9e6d9c4079adbefd29732680e3]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 20:16:44.356477 24997 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a177fdd3e9c489a9bafbfce528b6edd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:16:44.356652 25138 tablet_service.cc:1467] Processing CreateTablet for tablet 2a177fdd3e9c489a9bafbfce528b6edd (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=d354fd9e6d9c4079adbefd29732680e3]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 20:16:44.357167 25138 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a177fdd3e9c489a9bafbfce528b6edd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:16:44.358345 25278 tablet_service.cc:1467] Processing CreateTablet for tablet 2a177fdd3e9c489a9bafbfce528b6edd (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=d354fd9e6d9c4079adbefd29732680e3]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 20:16:44.358906 25278 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a177fdd3e9c489a9bafbfce528b6edd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:16:44.362022 25373 tablet_bootstrap.cc:492] T 2a177fdd3e9c489a9bafbfce528b6edd P 5a07c01f0567489fb51b6bb676a3764f: Bootstrap starting.
I20260620 20:16:44.363706 25373 tablet_bootstrap.cc:654] T 2a177fdd3e9c489a9bafbfce528b6edd P 5a07c01f0567489fb51b6bb676a3764f: Neither blocks nor log segments found. Creating new log.
I20260620 20:16:44.366060 25375 tablet_bootstrap.cc:492] T 2a177fdd3e9c489a9bafbfce528b6edd P 386309b3999d45769c3723f68adf1210: Bootstrap starting.
I20260620 20:16:44.367529 25375 tablet_bootstrap.cc:654] T 2a177fdd3e9c489a9bafbfce528b6edd P 386309b3999d45769c3723f68adf1210: Neither blocks nor log segments found. Creating new log.
I20260620 20:16:44.370616 25374 tablet_bootstrap.cc:492] T 2a177fdd3e9c489a9bafbfce528b6edd P a64d4977929340b7a5de6108995a78fa: Bootstrap starting.
I20260620 20:16:44.372802 25374 tablet_bootstrap.cc:654] T 2a177fdd3e9c489a9bafbfce528b6edd P a64d4977929340b7a5de6108995a78fa: Neither blocks nor log segments found. Creating new log.
I20260620 20:16:44.375998 25373 tablet_bootstrap.cc:492] T 2a177fdd3e9c489a9bafbfce528b6edd P 5a07c01f0567489fb51b6bb676a3764f: No bootstrap required, opened a new log
I20260620 20:16:44.376200 25373 ts_tablet_manager.cc:1397] T 2a177fdd3e9c489a9bafbfce528b6edd P 5a07c01f0567489fb51b6bb676a3764f: Time spent bootstrapping tablet: real 0.014s user 0.005s sys 0.007s
I20260620 20:16:44.376936 25375 tablet_bootstrap.cc:492] T 2a177fdd3e9c489a9bafbfce528b6edd P 386309b3999d45769c3723f68adf1210: No bootstrap required, opened a new log
I20260620 20:16:44.377103 25375 ts_tablet_manager.cc:1397] T 2a177fdd3e9c489a9bafbfce528b6edd P 386309b3999d45769c3723f68adf1210: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.004s
I20260620 20:16:44.377168 25373 raft_consensus.cc:348] T 2a177fdd3e9c489a9bafbfce528b6edd P 5a07c01f0567489fb51b6bb676a3764f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a07c01f0567489fb51b6bb676a3764f" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 43535 } } peers { permanent_uuid: "a64d4977929340b7a5de6108995a78fa" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 36825 } } peers { permanent_uuid: "386309b3999d45769c3723f68adf1210" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 44465 } }
I20260620 20:16:44.377506 25373 raft_consensus.cc:374] T 2a177fdd3e9c489a9bafbfce528b6edd P 5a07c01f0567489fb51b6bb676a3764f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:16:44.377609 25373 raft_consensus.cc:729] T 2a177fdd3e9c489a9bafbfce528b6edd P 5a07c01f0567489fb51b6bb676a3764f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a07c01f0567489fb51b6bb676a3764f, State: Initialized, Role: FOLLOWER
I20260620 20:16:44.377837 25375 raft_consensus.cc:348] T 2a177fdd3e9c489a9bafbfce528b6edd P 386309b3999d45769c3723f68adf1210 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a07c01f0567489fb51b6bb676a3764f" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 43535 } } peers { permanent_uuid: "a64d4977929340b7a5de6108995a78fa" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 36825 } } peers { permanent_uuid: "386309b3999d45769c3723f68adf1210" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 44465 } }
I20260620 20:16:44.377929 25373 consensus_queue.cc:260] T 2a177fdd3e9c489a9bafbfce528b6edd P 5a07c01f0567489fb51b6bb676a3764f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a07c01f0567489fb51b6bb676a3764f" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 43535 } } peers { permanent_uuid: "a64d4977929340b7a5de6108995a78fa" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 36825 } } peers { permanent_uuid: "386309b3999d45769c3723f68adf1210" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 44465 } }
I20260620 20:16:44.378108 25375 raft_consensus.cc:374] T 2a177fdd3e9c489a9bafbfce528b6edd P 386309b3999d45769c3723f68adf1210 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:16:44.378180 25375 raft_consensus.cc:729] T 2a177fdd3e9c489a9bafbfce528b6edd P 386309b3999d45769c3723f68adf1210 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 386309b3999d45769c3723f68adf1210, State: Initialized, Role: FOLLOWER
I20260620 20:16:44.378433 25394 tablet_replica.cc:439] T 2a177fdd3e9c489a9bafbfce528b6edd P 5a07c01f0567489fb51b6bb676a3764f: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a07c01f0567489fb51b6bb676a3764f" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 43535 } } peers { permanent_uuid: "a64d4977929340b7a5de6108995a78fa" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 36825 } } peers { permanent_uuid: "386309b3999d45769c3723f68adf1210" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 44465 } } }
I20260620 20:16:44.378796 25394 tablet_replica.cc:442] T 2a177fdd3e9c489a9bafbfce528b6edd P 5a07c01f0567489fb51b6bb676a3764f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 20:16:44.379402 25373 ts_tablet_manager.cc:1428] T 2a177fdd3e9c489a9bafbfce528b6edd P 5a07c01f0567489fb51b6bb676a3764f: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 20:16:44.380451 25375 consensus_queue.cc:260] T 2a177fdd3e9c489a9bafbfce528b6edd P 386309b3999d45769c3723f68adf1210 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a07c01f0567489fb51b6bb676a3764f" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 43535 } } peers { permanent_uuid: "a64d4977929340b7a5de6108995a78fa" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 36825 } } peers { permanent_uuid: "386309b3999d45769c3723f68adf1210" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 44465 } }
I20260620 20:16:44.380885 25380 tablet_replica.cc:439] T 2a177fdd3e9c489a9bafbfce528b6edd P 386309b3999d45769c3723f68adf1210: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a07c01f0567489fb51b6bb676a3764f" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 43535 } } peers { permanent_uuid: "a64d4977929340b7a5de6108995a78fa" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 36825 } } peers { permanent_uuid: "386309b3999d45769c3723f68adf1210" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 44465 } } }
I20260620 20:16:44.381224 25380 tablet_replica.cc:442] T 2a177fdd3e9c489a9bafbfce528b6edd P 386309b3999d45769c3723f68adf1210: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 20:16:44.382998 25375 ts_tablet_manager.cc:1428] T 2a177fdd3e9c489a9bafbfce528b6edd P 386309b3999d45769c3723f68adf1210: Time spent starting tablet: real 0.006s user 0.000s sys 0.002s
I20260620 20:16:44.386104 25374 tablet_bootstrap.cc:492] T 2a177fdd3e9c489a9bafbfce528b6edd P a64d4977929340b7a5de6108995a78fa: No bootstrap required, opened a new log
I20260620 20:16:44.386286 25374 ts_tablet_manager.cc:1397] T 2a177fdd3e9c489a9bafbfce528b6edd P a64d4977929340b7a5de6108995a78fa: Time spent bootstrapping tablet: real 0.016s user 0.006s sys 0.004s
I20260620 20:16:44.386868 25374 raft_consensus.cc:348] T 2a177fdd3e9c489a9bafbfce528b6edd P a64d4977929340b7a5de6108995a78fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a07c01f0567489fb51b6bb676a3764f" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 43535 } } peers { permanent_uuid: "a64d4977929340b7a5de6108995a78fa" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 36825 } } peers { permanent_uuid: "386309b3999d45769c3723f68adf1210" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 44465 } }
I20260620 20:16:44.387077 25374 raft_consensus.cc:374] T 2a177fdd3e9c489a9bafbfce528b6edd P a64d4977929340b7a5de6108995a78fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:16:44.387177 25374 raft_consensus.cc:729] T 2a177fdd3e9c489a9bafbfce528b6edd P a64d4977929340b7a5de6108995a78fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64d4977929340b7a5de6108995a78fa, State: Initialized, Role: FOLLOWER
I20260620 20:16:44.387400 25374 consensus_queue.cc:260] T 2a177fdd3e9c489a9bafbfce528b6edd P a64d4977929340b7a5de6108995a78fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a07c01f0567489fb51b6bb676a3764f" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 43535 } } peers { permanent_uuid: "a64d4977929340b7a5de6108995a78fa" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 36825 } } peers { permanent_uuid: "386309b3999d45769c3723f68adf1210" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 44465 } }
I20260620 20:16:44.387972 25374 ts_tablet_manager.cc:1428] T 2a177fdd3e9c489a9bafbfce528b6edd P a64d4977929340b7a5de6108995a78fa: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260620 20:16:44.387832 25379 tablet_replica.cc:439] T 2a177fdd3e9c489a9bafbfce528b6edd P a64d4977929340b7a5de6108995a78fa: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a07c01f0567489fb51b6bb676a3764f" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 43535 } } peers { permanent_uuid: "a64d4977929340b7a5de6108995a78fa" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 36825 } } peers { permanent_uuid: "386309b3999d45769c3723f68adf1210" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 44465 } } }
I20260620 20:16:44.388134 25379 tablet_replica.cc:442] T 2a177fdd3e9c489a9bafbfce528b6edd P a64d4977929340b7a5de6108995a78fa: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 20:16:44.483747 25394 raft_consensus.cc:482] T 2a177fdd3e9c489a9bafbfce528b6edd P 5a07c01f0567489fb51b6bb676a3764f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 20:16:44.484010 25394 raft_consensus.cc:504] T 2a177fdd3e9c489a9bafbfce528b6edd P 5a07c01f0567489fb51b6bb676a3764f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a07c01f0567489fb51b6bb676a3764f" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 43535 } } peers { permanent_uuid: "a64d4977929340b7a5de6108995a78fa" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 36825 } } peers { permanent_uuid: "386309b3999d45769c3723f68adf1210" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 44465 } }
I20260620 20:16:44.484885 25394 leader_election.cc:288] T 2a177fdd3e9c489a9bafbfce528b6edd P 5a07c01f0567489fb51b6bb676a3764f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a64d4977929340b7a5de6108995a78fa (127.16.217.2:36825), 386309b3999d45769c3723f68adf1210 (127.16.217.3:44465)
I20260620 20:16:44.485353 25158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a177fdd3e9c489a9bafbfce528b6edd" candidate_uuid: "5a07c01f0567489fb51b6bb676a3764f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64d4977929340b7a5de6108995a78fa" is_pre_election: true
I20260620 20:16:44.485466 25303 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a177fdd3e9c489a9bafbfce528b6edd" candidate_uuid: "5a07c01f0567489fb51b6bb676a3764f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "386309b3999d45769c3723f68adf1210" is_pre_election: true
I20260620 20:16:44.485747 25158 raft_consensus.cc:2454] T 2a177fdd3e9c489a9bafbfce528b6edd P a64d4977929340b7a5de6108995a78fa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5a07c01f0567489fb51b6bb676a3764f in term 0.
I20260620 20:16:44.485791 25303 raft_consensus.cc:2454] T 2a177fdd3e9c489a9bafbfce528b6edd P 386309b3999d45769c3723f68adf1210 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5a07c01f0567489fb51b6bb676a3764f in term 0.
I20260620 20:16:44.486276 24948 leader_election.cc:302] T 2a177fdd3e9c489a9bafbfce528b6edd P 5a07c01f0567489fb51b6bb676a3764f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 386309b3999d45769c3723f68adf1210, 5a07c01f0567489fb51b6bb676a3764f; no voters:
I20260620 20:16:44.486583 25394 raft_consensus.cc:2781] T 2a177fdd3e9c489a9bafbfce528b6edd P 5a07c01f0567489fb51b6bb676a3764f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 20:16:44.486698 25394 raft_consensus.cc:482] T 2a177fdd3e9c489a9bafbfce528b6edd P 5a07c01f0567489fb51b6bb676a3764f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 20:16:44.486857 25394 raft_consensus.cc:3037] T 2a177fdd3e9c489a9bafbfce528b6edd P 5a07c01f0567489fb51b6bb676a3764f [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:16:44.489271 25394 raft_consensus.cc:504] T 2a177fdd3e9c489a9bafbfce528b6edd P 5a07c01f0567489fb51b6bb676a3764f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a07c01f0567489fb51b6bb676a3764f" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 43535 } } peers { permanent_uuid: "a64d4977929340b7a5de6108995a78fa" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 36825 } } peers { permanent_uuid: "386309b3999d45769c3723f68adf1210" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 44465 } }
I20260620 20:16:44.490015 25394 leader_election.cc:288] T 2a177fdd3e9c489a9bafbfce528b6edd P 5a07c01f0567489fb51b6bb676a3764f [CANDIDATE]: Term 1 election: Requested vote from peers a64d4977929340b7a5de6108995a78fa (127.16.217.2:36825), 386309b3999d45769c3723f68adf1210 (127.16.217.3:44465)
I20260620 20:16:44.490350 25158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a177fdd3e9c489a9bafbfce528b6edd" candidate_uuid: "5a07c01f0567489fb51b6bb676a3764f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64d4977929340b7a5de6108995a78fa"
I20260620 20:16:44.490476 25303 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a177fdd3e9c489a9bafbfce528b6edd" candidate_uuid: "5a07c01f0567489fb51b6bb676a3764f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "386309b3999d45769c3723f68adf1210"
I20260620 20:16:44.490607 25158 raft_consensus.cc:3037] T 2a177fdd3e9c489a9bafbfce528b6edd P a64d4977929340b7a5de6108995a78fa [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:16:44.490721 25303 raft_consensus.cc:3037] T 2a177fdd3e9c489a9bafbfce528b6edd P 386309b3999d45769c3723f68adf1210 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:16:44.492769 25158 raft_consensus.cc:2454] T 2a177fdd3e9c489a9bafbfce528b6edd P a64d4977929340b7a5de6108995a78fa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a07c01f0567489fb51b6bb676a3764f in term 1.
I20260620 20:16:44.493198 24948 leader_election.cc:302] T 2a177fdd3e9c489a9bafbfce528b6edd P 5a07c01f0567489fb51b6bb676a3764f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a07c01f0567489fb51b6bb676a3764f, a64d4977929340b7a5de6108995a78fa; no voters:
I20260620 20:16:44.493507 25394 raft_consensus.cc:2781] T 2a177fdd3e9c489a9bafbfce528b6edd P 5a07c01f0567489fb51b6bb676a3764f [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:16:44.493695 25303 raft_consensus.cc:2454] T 2a177fdd3e9c489a9bafbfce528b6edd P 386309b3999d45769c3723f68adf1210 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a07c01f0567489fb51b6bb676a3764f in term 1.
I20260620 20:16:44.494333 25394 raft_consensus.cc:686] T 2a177fdd3e9c489a9bafbfce528b6edd P 5a07c01f0567489fb51b6bb676a3764f [term 1 LEADER]: Becoming Leader. State: Replica: 5a07c01f0567489fb51b6bb676a3764f, State: Running, Role: LEADER
I20260620 20:16:44.494607 25394 consensus_queue.cc:237] T 2a177fdd3e9c489a9bafbfce528b6edd P 5a07c01f0567489fb51b6bb676a3764f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a07c01f0567489fb51b6bb676a3764f" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 43535 } } peers { permanent_uuid: "a64d4977929340b7a5de6108995a78fa" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 36825 } } peers { permanent_uuid: "386309b3999d45769c3723f68adf1210" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 44465 } }
I20260620 20:16:44.496229 25381 tablet_replica.cc:439] T 2a177fdd3e9c489a9bafbfce528b6edd P 5a07c01f0567489fb51b6bb676a3764f: TxnStatusTablet state changed. Reason: New leader 5a07c01f0567489fb51b6bb676a3764f. Latest consensus state: current_term: 1 leader_uuid: "5a07c01f0567489fb51b6bb676a3764f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a07c01f0567489fb51b6bb676a3764f" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 43535 } } peers { permanent_uuid: "a64d4977929340b7a5de6108995a78fa" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 36825 } } peers { permanent_uuid: "386309b3999d45769c3723f68adf1210" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 44465 } } }
I20260620 20:16:44.496522 25381 tablet_replica.cc:442] T 2a177fdd3e9c489a9bafbfce528b6edd P 5a07c01f0567489fb51b6bb676a3764f: This TxnStatusTablet replica's current role is: LEADER
I20260620 20:16:44.498793 24860 catalog_manager.cc:5495] T 2a177fdd3e9c489a9bafbfce528b6edd P 5a07c01f0567489fb51b6bb676a3764f reported cstate change: term changed from 0 to 1, leader changed from <none> to 5a07c01f0567489fb51b6bb676a3764f (127.16.217.1). New cstate: current_term: 1 leader_uuid: "5a07c01f0567489fb51b6bb676a3764f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a07c01f0567489fb51b6bb676a3764f" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 43535 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a64d4977929340b7a5de6108995a78fa" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 36825 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "386309b3999d45769c3723f68adf1210" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 44465 } health_report { overall_health: UNKNOWN } } }
I20260620 20:16:44.500727 25417 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260620 20:16:44.500989 25417 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260620 20:16:44.528162 25370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:44.522889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38022 (local address 127.16.217.1:43535)
0620 20:16:44.523096 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0620 20:16:44.523103 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:16:44.523127 (+ 24us) server_negotiation.cc:407] Connection header received
0620 20:16:44.523411 (+ 284us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:16:44.523419 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:16:44.523512 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:16:44.523641 (+ 129us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 20:16:44.524063 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:44.524963 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:44.526364 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:44.526819 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:44.526963 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:44.527371 (+ 408us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 20:16:44.527472 (+ 101us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 20:16:44.527583 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0620 20:16:44.527766 (+ 183us) server_negotiation.cc:299] Negotiation successful
0620 20:16:44.527846 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260620 20:16:44.528184 25411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:44.522701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.1:43535 (local address 127.0.0.1:38022)
0620 20:16:44.523016 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0620 20:16:44.523039 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 20:16:44.523208 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:16:44.523681 (+ 473us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:16:44.523687 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:16:44.523701 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 20:16:44.523908 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:16:44.523917 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:16:44.525175 (+ 1258us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:16:44.525183 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:16:44.526233 (+ 1050us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:16:44.526244 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:16:44.526975 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:16:44.526980 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:16:44.527146 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:44.527199 (+ 53us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 20:16:44.527593 (+ 394us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 20:16:44.527621 (+ 28us) client_negotiation.cc:769] Sending connection context
0620 20:16:44.527741 (+ 120us) client_negotiation.cc:240] Negotiation successful
0620 20:16:44.527884 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":84}
I20260620 20:16:44.531965 25303 raft_consensus.cc:1261] T 2a177fdd3e9c489a9bafbfce528b6edd P 386309b3999d45769c3723f68adf1210 [term 1 FOLLOWER]: Refusing update from remote peer 5a07c01f0567489fb51b6bb676a3764f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:16:44.532101 25158 raft_consensus.cc:1261] T 2a177fdd3e9c489a9bafbfce528b6edd P a64d4977929340b7a5de6108995a78fa [term 1 FOLLOWER]: Refusing update from remote peer 5a07c01f0567489fb51b6bb676a3764f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:16:44.532686 25381 consensus_queue.cc:1035] T 2a177fdd3e9c489a9bafbfce528b6edd P 5a07c01f0567489fb51b6bb676a3764f [LEADER]: Connected to new peer: Peer: permanent_uuid: "386309b3999d45769c3723f68adf1210" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 44465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:16:44.533252 25394 consensus_queue.cc:1035] T 2a177fdd3e9c489a9bafbfce528b6edd P 5a07c01f0567489fb51b6bb676a3764f [LEADER]: Connected to new peer: Peer: permanent_uuid: "a64d4977929340b7a5de6108995a78fa" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 36825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:16:44.536239 25380 tablet_replica.cc:439] T 2a177fdd3e9c489a9bafbfce528b6edd P 386309b3999d45769c3723f68adf1210: TxnStatusTablet state changed. Reason: New leader 5a07c01f0567489fb51b6bb676a3764f. Latest consensus state: current_term: 1 leader_uuid: "5a07c01f0567489fb51b6bb676a3764f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a07c01f0567489fb51b6bb676a3764f" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 43535 } } peers { permanent_uuid: "a64d4977929340b7a5de6108995a78fa" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 36825 } } peers { permanent_uuid: "386309b3999d45769c3723f68adf1210" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 44465 } } }
I20260620 20:16:44.536491 25380 tablet_replica.cc:442] T 2a177fdd3e9c489a9bafbfce528b6edd P 386309b3999d45769c3723f68adf1210: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 20:16:44.538080 25381 tablet_replica.cc:439] T 2a177fdd3e9c489a9bafbfce528b6edd P 5a07c01f0567489fb51b6bb676a3764f: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5a07c01f0567489fb51b6bb676a3764f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a07c01f0567489fb51b6bb676a3764f" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 43535 } } peers { permanent_uuid: "a64d4977929340b7a5de6108995a78fa" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 36825 } } peers { permanent_uuid: "386309b3999d45769c3723f68adf1210" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 44465 } } }
I20260620 20:16:44.538327 25381 tablet_replica.cc:442] T 2a177fdd3e9c489a9bafbfce528b6edd P 5a07c01f0567489fb51b6bb676a3764f: This TxnStatusTablet replica's current role is: LEADER
I20260620 20:16:44.541673 25380 tablet_replica.cc:439] T 2a177fdd3e9c489a9bafbfce528b6edd P 386309b3999d45769c3723f68adf1210: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5a07c01f0567489fb51b6bb676a3764f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a07c01f0567489fb51b6bb676a3764f" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 43535 } } peers { permanent_uuid: "a64d4977929340b7a5de6108995a78fa" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 36825 } } peers { permanent_uuid: "386309b3999d45769c3723f68adf1210" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 44465 } } }
I20260620 20:16:44.541962 25380 tablet_replica.cc:442] T 2a177fdd3e9c489a9bafbfce528b6edd P 386309b3999d45769c3723f68adf1210: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 20:16:44.546988 25379 tablet_replica.cc:439] T 2a177fdd3e9c489a9bafbfce528b6edd P a64d4977929340b7a5de6108995a78fa: TxnStatusTablet state changed. Reason: New leader 5a07c01f0567489fb51b6bb676a3764f. Latest consensus state: current_term: 1 leader_uuid: "5a07c01f0567489fb51b6bb676a3764f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a07c01f0567489fb51b6bb676a3764f" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 43535 } } peers { permanent_uuid: "a64d4977929340b7a5de6108995a78fa" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 36825 } } peers { permanent_uuid: "386309b3999d45769c3723f68adf1210" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 44465 } } }
I20260620 20:16:44.547281 25379 tablet_replica.cc:442] T 2a177fdd3e9c489a9bafbfce528b6edd P a64d4977929340b7a5de6108995a78fa: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 20:16:44.548794 25386 tablet_replica.cc:439] T 2a177fdd3e9c489a9bafbfce528b6edd P 5a07c01f0567489fb51b6bb676a3764f: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5a07c01f0567489fb51b6bb676a3764f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a07c01f0567489fb51b6bb676a3764f" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 43535 } } peers { permanent_uuid: "a64d4977929340b7a5de6108995a78fa" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 36825 } } peers { permanent_uuid: "386309b3999d45769c3723f68adf1210" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 44465 } } }
I20260620 20:16:44.549106 25386 tablet_replica.cc:442] T 2a177fdd3e9c489a9bafbfce528b6edd P 5a07c01f0567489fb51b6bb676a3764f: This TxnStatusTablet replica's current role is: LEADER
I20260620 20:16:44.553480 25379 tablet_replica.cc:439] T 2a177fdd3e9c489a9bafbfce528b6edd P a64d4977929340b7a5de6108995a78fa: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5a07c01f0567489fb51b6bb676a3764f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a07c01f0567489fb51b6bb676a3764f" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 43535 } } peers { permanent_uuid: "a64d4977929340b7a5de6108995a78fa" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 36825 } } peers { permanent_uuid: "386309b3999d45769c3723f68adf1210" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 44465 } } }
I20260620 20:16:44.553725 25379 tablet_replica.cc:442] T 2a177fdd3e9c489a9bafbfce528b6edd P a64d4977929340b7a5de6108995a78fa: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 20:16:44.571382 25370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:44.563207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.217.1:33039 (local address 127.16.217.1:43535)
0620 20:16:44.563414 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0620 20:16:44.563422 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 20:16:44.566444 (+ 3022us) server_negotiation.cc:407] Connection header received
0620 20:16:44.566649 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:16:44.566654 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:16:44.566735 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:16:44.566817 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 20:16:44.567323 (+ 506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:44.568125 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:44.569515 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:44.570054 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:44.570189 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:44.570601 (+ 412us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 20:16:44.570734 (+ 133us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 20:16:44.570827 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0620 20:16:44.570949 (+ 122us) server_negotiation.cc:299] Negotiation successful
0620 20:16:44.571037 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":20,"server-negotiator.queue_time_us":48}
I20260620 20:16:44.571496 25423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:44.563172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.1:43535 (local address 127.16.217.1:33039)
0620 20:16:44.566348 (+ 3176us) negotiation.cc:107] Waiting for socket to connect
0620 20:16:44.566364 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 20:16:44.566514 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:16:44.566839 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:16:44.566843 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:16:44.566852 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 20:16:44.567176 (+ 324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:16:44.567183 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:16:44.568245 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:16:44.568252 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:16:44.569386 (+ 1134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:16:44.569395 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:16:44.570238 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:16:44.570246 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:16:44.570402 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:44.570449 (+ 47us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 20:16:44.570824 (+ 375us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 20:16:44.570834 (+ 10us) client_negotiation.cc:769] Sending connection context
0620 20:16:44.570933 (+ 99us) client_negotiation.cc:240] Negotiation successful
0620 20:16:44.571070 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3025,"mutex_wait_us":1,"thread_start_us":2887,"threads_started":1}
I20260620 20:16:44.581029 25303 raft_consensus.cc:1261] T 28eab8dc853f47ffac0efd6edc989abe P 386309b3999d45769c3723f68adf1210 [term 1 FOLLOWER]: Refusing update from remote peer 5a07c01f0567489fb51b6bb676a3764f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:16:44.581305 25158 raft_consensus.cc:1261] T 28eab8dc853f47ffac0efd6edc989abe P a64d4977929340b7a5de6108995a78fa [term 1 FOLLOWER]: Refusing update from remote peer 5a07c01f0567489fb51b6bb676a3764f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:16:44.581882 25386 consensus_queue.cc:1035] T 28eab8dc853f47ffac0efd6edc989abe P 5a07c01f0567489fb51b6bb676a3764f [LEADER]: Connected to new peer: Peer: permanent_uuid: "386309b3999d45769c3723f68adf1210" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 44465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:16:44.582304 25386 consensus_queue.cc:1035] T 28eab8dc853f47ffac0efd6edc989abe P 5a07c01f0567489fb51b6bb676a3764f [LEADER]: Connected to new peer: Peer: permanent_uuid: "a64d4977929340b7a5de6108995a78fa" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 36825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:16:44.654079 24860 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40564:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 20:16:44.654325 24860 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40564:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 20:16:44.657208 24860 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 827ffa1470674fdfa9e771771e6a7d94: Sending DeleteTablet for 3 replicas of tablet 28eab8dc853f47ffac0efd6edc989abe
I20260620 20:16:44.658026 24998 tablet_service.cc:1514] Processing DeleteTablet for tablet 28eab8dc853f47ffac0efd6edc989abe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:16:44 UTC) from {username='oryx', principal='oryx/127.16.217.62@KRBTEST.COM'} at 127.0.0.1:38004
I20260620 20:16:44.658100 25138 tablet_service.cc:1514] Processing DeleteTablet for tablet 28eab8dc853f47ffac0efd6edc989abe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:16:44 UTC) from {username='oryx', principal='oryx/127.16.217.62@KRBTEST.COM'} at 127.0.0.1:38734
I20260620 20:16:44.658293 25400 tablet_replica.cc:330] T 28eab8dc853f47ffac0efd6edc989abe P 5a07c01f0567489fb51b6bb676a3764f: stopping tablet replica
I20260620 20:16:44.658262 25278 tablet_service.cc:1514] Processing DeleteTablet for tablet 28eab8dc853f47ffac0efd6edc989abe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:16:44 UTC) from {username='oryx', principal='oryx/127.16.217.62@KRBTEST.COM'} at 127.0.0.1:49776
I20260620 20:16:44.658365 25399 tablet_replica.cc:330] T 28eab8dc853f47ffac0efd6edc989abe P a64d4977929340b7a5de6108995a78fa: stopping tablet replica
I20260620 20:16:44.658495 25400 raft_consensus.cc:2229] T 28eab8dc853f47ffac0efd6edc989abe P 5a07c01f0567489fb51b6bb676a3764f [term 1 LEADER]: Raft consensus shutting down.
I20260620 20:16:44.658511 25401 tablet_replica.cc:330] T 28eab8dc853f47ffac0efd6edc989abe P 386309b3999d45769c3723f68adf1210: stopping tablet replica
I20260620 20:16:44.658572 25399 raft_consensus.cc:2229] T 28eab8dc853f47ffac0efd6edc989abe P a64d4977929340b7a5de6108995a78fa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 20:16:44.658680 25401 raft_consensus.cc:2229] T 28eab8dc853f47ffac0efd6edc989abe P 386309b3999d45769c3723f68adf1210 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 20:16:44.658751 25399 raft_consensus.cc:2258] T 28eab8dc853f47ffac0efd6edc989abe P a64d4977929340b7a5de6108995a78fa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:16:44.658795 25400 raft_consensus.cc:2258] T 28eab8dc853f47ffac0efd6edc989abe P 5a07c01f0567489fb51b6bb676a3764f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:16:44.658843 25401 raft_consensus.cc:2258] T 28eab8dc853f47ffac0efd6edc989abe P 386309b3999d45769c3723f68adf1210 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:16:44.660281 25401 ts_tablet_manager.cc:1905] T 28eab8dc853f47ffac0efd6edc989abe P 386309b3999d45769c3723f68adf1210: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:16:44.662953 25400 ts_tablet_manager.cc:1905] T 28eab8dc853f47ffac0efd6edc989abe P 5a07c01f0567489fb51b6bb676a3764f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:16:44.663071 25399 ts_tablet_manager.cc:1905] T 28eab8dc853f47ffac0efd6edc989abe P a64d4977929340b7a5de6108995a78fa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:16:44.667130 25401 ts_tablet_manager.cc:1918] T 28eab8dc853f47ffac0efd6edc989abe P 386309b3999d45769c3723f68adf1210: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 20:16:44.667282 25401 log.cc:1196] T 28eab8dc853f47ffac0efd6edc989abe P 386309b3999d45769c3723f68adf1210: Deleting WAL directory at /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986536687152-17252-0/minicluster-data/ts-2/wal/wals/28eab8dc853f47ffac0efd6edc989abe
I20260620 20:16:44.667687 25401 ts_tablet_manager.cc:1939] T 28eab8dc853f47ffac0efd6edc989abe P 386309b3999d45769c3723f68adf1210: Deleting consensus metadata
I20260620 20:16:44.668797 24843 catalog_manager.cc:4841] TS 386309b3999d45769c3723f68adf1210 (127.16.217.3:44465): tablet 28eab8dc853f47ffac0efd6edc989abe (table test-table [id=d54f0c3ee18643eaaa5581dbbd6cad98]) successfully deleted
I20260620 20:16:44.669770 25399 ts_tablet_manager.cc:1918] T 28eab8dc853f47ffac0efd6edc989abe P a64d4977929340b7a5de6108995a78fa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 20:16:44.670387 25400 ts_tablet_manager.cc:1918] T 28eab8dc853f47ffac0efd6edc989abe P 5a07c01f0567489fb51b6bb676a3764f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 20:16:44.670552 25400 log.cc:1196] T 28eab8dc853f47ffac0efd6edc989abe P 5a07c01f0567489fb51b6bb676a3764f: Deleting WAL directory at /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986536687152-17252-0/minicluster-data/ts-0/wal/wals/28eab8dc853f47ffac0efd6edc989abe
I20260620 20:16:44.670998 25400 ts_tablet_manager.cc:1939] T 28eab8dc853f47ffac0efd6edc989abe P 5a07c01f0567489fb51b6bb676a3764f: Deleting consensus metadata
I20260620 20:16:44.672137 24846 catalog_manager.cc:4841] TS 5a07c01f0567489fb51b6bb676a3764f (127.16.217.1:43535): tablet 28eab8dc853f47ffac0efd6edc989abe (table test-table [id=d54f0c3ee18643eaaa5581dbbd6cad98]) successfully deleted
I20260620 20:16:44.674607 25399 log.cc:1196] T 28eab8dc853f47ffac0efd6edc989abe P a64d4977929340b7a5de6108995a78fa: Deleting WAL directory at /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986536687152-17252-0/minicluster-data/ts-1/wal/wals/28eab8dc853f47ffac0efd6edc989abe
I20260620 20:16:44.675176 25399 ts_tablet_manager.cc:1939] T 28eab8dc853f47ffac0efd6edc989abe P a64d4977929340b7a5de6108995a78fa: Deleting consensus metadata
I20260620 20:16:44.676581 24843 catalog_manager.cc:4841] TS a64d4977929340b7a5de6108995a78fa (127.16.217.2:36825): tablet 28eab8dc853f47ffac0efd6edc989abe (table test-table [id=d54f0c3ee18643eaaa5581dbbd6cad98]) successfully deleted
Jun 20 20:16:44 dist-test-slave-2hpg krb5kdc[24814](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986604, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 20:16:44 dist-test-slave-2hpg krb5kdc[24814](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.16.217.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 20:16:44 dist-test-slave-2hpg krb5kdc[24814](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.16.217.62@KRBTEST.COM, Server not found in Kerberos database
W20260620 20:16:44.722743 25412 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:16:44.710404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40580 (local address 127.16.217.62:39789)
0620 20:16:44.710686 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0620 20:16:44.710693 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:16:44.711032 (+ 339us) server_negotiation.cc:407] Connection header received
0620 20:16:44.711229 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:16:44.711233 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:16:44.711320 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:16:44.711415 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:16:44.713054 (+ 1639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:44.713905 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:44.715887 (+ 1982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:44.716319 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:44.716416 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:44.722481 (+ 6065us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40580: BlockingRecv error: recv got EOF from 127.0.0.1:40580 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
Jun 20 20:16:44 dist-test-slave-2hpg krb5kdc[24814](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986604, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.16.217.62@KRBTEST.COM
I20260620 20:16:44.744133 25412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:44.725395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40594 (local address 127.16.217.62:39789)
0620 20:16:44.725567 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0620 20:16:44.725573 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:16:44.725836 (+ 263us) server_negotiation.cc:407] Connection header received
0620 20:16:44.726029 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:16:44.726035 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:16:44.726110 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:16:44.726226 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:16:44.727347 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:44.728468 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:44.729657 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:44.730328 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:44.730454 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:44.736709 (+ 6255us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:16:44.736715 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:16:44.736721 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:16:44.736761 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:16:44.740334 (+ 3573us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:16:44.741493 (+ 1159us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:16:44.741499 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:16:44.741503 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:16:44.741647 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:16:44.742104 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:16:44.742110 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:16:44.742114 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:16:44.742481 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:16:44.742745 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0620 20:16:44.743278 (+ 533us) server_negotiation.cc:299] Negotiation successful
0620 20:16:44.743538 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 20:16:44.749424 24860 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40594:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 20:16:44.750284 24860 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 20:16:44.761719 24998 tablet_service.cc:1467] Processing CreateTablet for tablet 485abc1387924f4aa537abe9bd114c9f (DEFAULT_TABLE table=test-table [id=e2de26cd60424214be14e198b28ce757]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:16:44.761902 25138 tablet_service.cc:1467] Processing CreateTablet for tablet 485abc1387924f4aa537abe9bd114c9f (DEFAULT_TABLE table=test-table [id=e2de26cd60424214be14e198b28ce757]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:16:44.761902 25278 tablet_service.cc:1467] Processing CreateTablet for tablet 485abc1387924f4aa537abe9bd114c9f (DEFAULT_TABLE table=test-table [id=e2de26cd60424214be14e198b28ce757]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:16:44.762228 24998 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 485abc1387924f4aa537abe9bd114c9f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:16:44.762302 25278 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 485abc1387924f4aa537abe9bd114c9f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:16:44.762302 25138 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 485abc1387924f4aa537abe9bd114c9f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:16:44.768550 25375 tablet_bootstrap.cc:492] T 485abc1387924f4aa537abe9bd114c9f P 386309b3999d45769c3723f68adf1210: Bootstrap starting.
I20260620 20:16:44.768548 25374 tablet_bootstrap.cc:492] T 485abc1387924f4aa537abe9bd114c9f P a64d4977929340b7a5de6108995a78fa: Bootstrap starting.
I20260620 20:16:44.770147 25374 tablet_bootstrap.cc:654] T 485abc1387924f4aa537abe9bd114c9f P a64d4977929340b7a5de6108995a78fa: Neither blocks nor log segments found. Creating new log.
I20260620 20:16:44.770149 25375 tablet_bootstrap.cc:654] T 485abc1387924f4aa537abe9bd114c9f P 386309b3999d45769c3723f68adf1210: Neither blocks nor log segments found. Creating new log.
I20260620 20:16:44.770365 25373 tablet_bootstrap.cc:492] T 485abc1387924f4aa537abe9bd114c9f P 5a07c01f0567489fb51b6bb676a3764f: Bootstrap starting.
I20260620 20:16:44.772065 25373 tablet_bootstrap.cc:654] T 485abc1387924f4aa537abe9bd114c9f P 5a07c01f0567489fb51b6bb676a3764f: Neither blocks nor log segments found. Creating new log.
I20260620 20:16:44.781832 25375 tablet_bootstrap.cc:492] T 485abc1387924f4aa537abe9bd114c9f P 386309b3999d45769c3723f68adf1210: No bootstrap required, opened a new log
I20260620 20:16:44.782037 25375 ts_tablet_manager.cc:1397] T 485abc1387924f4aa537abe9bd114c9f P 386309b3999d45769c3723f68adf1210: Time spent bootstrapping tablet: real 0.014s user 0.010s sys 0.001s
I20260620 20:16:44.782884 25375 raft_consensus.cc:348] T 485abc1387924f4aa537abe9bd114c9f P 386309b3999d45769c3723f68adf1210 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a07c01f0567489fb51b6bb676a3764f" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 43535 } } peers { permanent_uuid: "a64d4977929340b7a5de6108995a78fa" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 36825 } } peers { permanent_uuid: "386309b3999d45769c3723f68adf1210" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 44465 } }
I20260620 20:16:44.783151 25374 tablet_bootstrap.cc:492] T 485abc1387924f4aa537abe9bd114c9f P a64d4977929340b7a5de6108995a78fa: No bootstrap required, opened a new log
I20260620 20:16:44.783147 25375 raft_consensus.cc:374] T 485abc1387924f4aa537abe9bd114c9f P 386309b3999d45769c3723f68adf1210 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:16:44.783252 25375 raft_consensus.cc:729] T 485abc1387924f4aa537abe9bd114c9f P 386309b3999d45769c3723f68adf1210 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 386309b3999d45769c3723f68adf1210, State: Initialized, Role: FOLLOWER
I20260620 20:16:44.783293 25374 ts_tablet_manager.cc:1397] T 485abc1387924f4aa537abe9bd114c9f P a64d4977929340b7a5de6108995a78fa: Time spent bootstrapping tablet: real 0.015s user 0.000s sys 0.004s
I20260620 20:16:44.783541 25375 consensus_queue.cc:260] T 485abc1387924f4aa537abe9bd114c9f P 386309b3999d45769c3723f68adf1210 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a07c01f0567489fb51b6bb676a3764f" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 43535 } } peers { permanent_uuid: "a64d4977929340b7a5de6108995a78fa" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 36825 } } peers { permanent_uuid: "386309b3999d45769c3723f68adf1210" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 44465 } }
I20260620 20:16:44.784049 25374 raft_consensus.cc:348] T 485abc1387924f4aa537abe9bd114c9f P a64d4977929340b7a5de6108995a78fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a07c01f0567489fb51b6bb676a3764f" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 43535 } } peers { permanent_uuid: "a64d4977929340b7a5de6108995a78fa" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 36825 } } peers { permanent_uuid: "386309b3999d45769c3723f68adf1210" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 44465 } }
I20260620 20:16:44.784299 25374 raft_consensus.cc:374] T 485abc1387924f4aa537abe9bd114c9f P a64d4977929340b7a5de6108995a78fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:16:44.784405 25375 ts_tablet_manager.cc:1428] T 485abc1387924f4aa537abe9bd114c9f P 386309b3999d45769c3723f68adf1210: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260620 20:16:44.784406 25374 raft_consensus.cc:729] T 485abc1387924f4aa537abe9bd114c9f P a64d4977929340b7a5de6108995a78fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64d4977929340b7a5de6108995a78fa, State: Initialized, Role: FOLLOWER
I20260620 20:16:44.784610 25374 consensus_queue.cc:260] T 485abc1387924f4aa537abe9bd114c9f P a64d4977929340b7a5de6108995a78fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a07c01f0567489fb51b6bb676a3764f" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 43535 } } peers { permanent_uuid: "a64d4977929340b7a5de6108995a78fa" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 36825 } } peers { permanent_uuid: "386309b3999d45769c3723f68adf1210" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 44465 } }
I20260620 20:16:44.785079 25374 ts_tablet_manager.cc:1428] T 485abc1387924f4aa537abe9bd114c9f P a64d4977929340b7a5de6108995a78fa: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260620 20:16:44.791921 25373 tablet_bootstrap.cc:492] T 485abc1387924f4aa537abe9bd114c9f P 5a07c01f0567489fb51b6bb676a3764f: No bootstrap required, opened a new log
I20260620 20:16:44.792101 25373 ts_tablet_manager.cc:1397] T 485abc1387924f4aa537abe9bd114c9f P 5a07c01f0567489fb51b6bb676a3764f: Time spent bootstrapping tablet: real 0.022s user 0.008s sys 0.002s
I20260620 20:16:44.792881 25373 raft_consensus.cc:348] T 485abc1387924f4aa537abe9bd114c9f P 5a07c01f0567489fb51b6bb676a3764f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a07c01f0567489fb51b6bb676a3764f" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 43535 } } peers { permanent_uuid: "a64d4977929340b7a5de6108995a78fa" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 36825 } } peers { permanent_uuid: "386309b3999d45769c3723f68adf1210" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 44465 } }
I20260620 20:16:44.793169 25373 raft_consensus.cc:374] T 485abc1387924f4aa537abe9bd114c9f P 5a07c01f0567489fb51b6bb676a3764f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:16:44.793296 25373 raft_consensus.cc:729] T 485abc1387924f4aa537abe9bd114c9f P 5a07c01f0567489fb51b6bb676a3764f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a07c01f0567489fb51b6bb676a3764f, State: Initialized, Role: FOLLOWER
I20260620 20:16:44.793591 25373 consensus_queue.cc:260] T 485abc1387924f4aa537abe9bd114c9f P 5a07c01f0567489fb51b6bb676a3764f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a07c01f0567489fb51b6bb676a3764f" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 43535 } } peers { permanent_uuid: "a64d4977929340b7a5de6108995a78fa" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 36825 } } peers { permanent_uuid: "386309b3999d45769c3723f68adf1210" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 44465 } }
I20260620 20:16:44.794284 25373 ts_tablet_manager.cc:1428] T 485abc1387924f4aa537abe9bd114c9f P 5a07c01f0567489fb51b6bb676a3764f: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20260620 20:16:44.868132 25386 raft_consensus.cc:482] T 485abc1387924f4aa537abe9bd114c9f P 5a07c01f0567489fb51b6bb676a3764f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 20:16:44.868427 25386 raft_consensus.cc:504] T 485abc1387924f4aa537abe9bd114c9f P 5a07c01f0567489fb51b6bb676a3764f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a07c01f0567489fb51b6bb676a3764f" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 43535 } } peers { permanent_uuid: "a64d4977929340b7a5de6108995a78fa" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 36825 } } peers { permanent_uuid: "386309b3999d45769c3723f68adf1210" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 44465 } }
I20260620 20:16:44.869019 25386 leader_election.cc:288] T 485abc1387924f4aa537abe9bd114c9f P 5a07c01f0567489fb51b6bb676a3764f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a64d4977929340b7a5de6108995a78fa (127.16.217.2:36825), 386309b3999d45769c3723f68adf1210 (127.16.217.3:44465)
I20260620 20:16:44.869676 25301 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "485abc1387924f4aa537abe9bd114c9f" candidate_uuid: "5a07c01f0567489fb51b6bb676a3764f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "386309b3999d45769c3723f68adf1210" is_pre_election: true
I20260620 20:16:44.869709 25157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "485abc1387924f4aa537abe9bd114c9f" candidate_uuid: "5a07c01f0567489fb51b6bb676a3764f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64d4977929340b7a5de6108995a78fa" is_pre_election: true
I20260620 20:16:44.869954 25301 raft_consensus.cc:2454] T 485abc1387924f4aa537abe9bd114c9f P 386309b3999d45769c3723f68adf1210 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5a07c01f0567489fb51b6bb676a3764f in term 0.
I20260620 20:16:44.869998 25157 raft_consensus.cc:2454] T 485abc1387924f4aa537abe9bd114c9f P a64d4977929340b7a5de6108995a78fa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5a07c01f0567489fb51b6bb676a3764f in term 0.
I20260620 20:16:44.870437 24948 leader_election.cc:302] T 485abc1387924f4aa537abe9bd114c9f P 5a07c01f0567489fb51b6bb676a3764f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 386309b3999d45769c3723f68adf1210, 5a07c01f0567489fb51b6bb676a3764f; no voters:
I20260620 20:16:44.870745 25386 raft_consensus.cc:2781] T 485abc1387924f4aa537abe9bd114c9f P 5a07c01f0567489fb51b6bb676a3764f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 20:16:44.870865 25386 raft_consensus.cc:482] T 485abc1387924f4aa537abe9bd114c9f P 5a07c01f0567489fb51b6bb676a3764f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 20:16:44.870991 25386 raft_consensus.cc:3037] T 485abc1387924f4aa537abe9bd114c9f P 5a07c01f0567489fb51b6bb676a3764f [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:16:44.873296 25386 raft_consensus.cc:504] T 485abc1387924f4aa537abe9bd114c9f P 5a07c01f0567489fb51b6bb676a3764f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a07c01f0567489fb51b6bb676a3764f" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 43535 } } peers { permanent_uuid: "a64d4977929340b7a5de6108995a78fa" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 36825 } } peers { permanent_uuid: "386309b3999d45769c3723f68adf1210" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 44465 } }
I20260620 20:16:44.873847 25386 leader_election.cc:288] T 485abc1387924f4aa537abe9bd114c9f P 5a07c01f0567489fb51b6bb676a3764f [CANDIDATE]: Term 1 election: Requested vote from peers a64d4977929340b7a5de6108995a78fa (127.16.217.2:36825), 386309b3999d45769c3723f68adf1210 (127.16.217.3:44465)
I20260620 20:16:44.874380 25157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "485abc1387924f4aa537abe9bd114c9f" candidate_uuid: "5a07c01f0567489fb51b6bb676a3764f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64d4977929340b7a5de6108995a78fa"
I20260620 20:16:44.874496 25301 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "485abc1387924f4aa537abe9bd114c9f" candidate_uuid: "5a07c01f0567489fb51b6bb676a3764f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "386309b3999d45769c3723f68adf1210"
I20260620 20:16:44.874598 25157 raft_consensus.cc:3037] T 485abc1387924f4aa537abe9bd114c9f P a64d4977929340b7a5de6108995a78fa [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:16:44.874679 25301 raft_consensus.cc:3037] T 485abc1387924f4aa537abe9bd114c9f P 386309b3999d45769c3723f68adf1210 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:16:44.877625 25301 raft_consensus.cc:2454] T 485abc1387924f4aa537abe9bd114c9f P 386309b3999d45769c3723f68adf1210 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a07c01f0567489fb51b6bb676a3764f in term 1.
I20260620 20:16:44.877626 25157 raft_consensus.cc:2454] T 485abc1387924f4aa537abe9bd114c9f P a64d4977929340b7a5de6108995a78fa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a07c01f0567489fb51b6bb676a3764f in term 1.
I20260620 20:16:44.878119 24948 leader_election.cc:302] T 485abc1387924f4aa537abe9bd114c9f P 5a07c01f0567489fb51b6bb676a3764f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a07c01f0567489fb51b6bb676a3764f, a64d4977929340b7a5de6108995a78fa; no voters:
I20260620 20:16:44.878424 25386 raft_consensus.cc:2781] T 485abc1387924f4aa537abe9bd114c9f P 5a07c01f0567489fb51b6bb676a3764f [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:16:44.878664 25386 raft_consensus.cc:686] T 485abc1387924f4aa537abe9bd114c9f P 5a07c01f0567489fb51b6bb676a3764f [term 1 LEADER]: Becoming Leader. State: Replica: 5a07c01f0567489fb51b6bb676a3764f, State: Running, Role: LEADER
I20260620 20:16:44.878930 25386 consensus_queue.cc:237] T 485abc1387924f4aa537abe9bd114c9f P 5a07c01f0567489fb51b6bb676a3764f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a07c01f0567489fb51b6bb676a3764f" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 43535 } } peers { permanent_uuid: "a64d4977929340b7a5de6108995a78fa" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 36825 } } peers { permanent_uuid: "386309b3999d45769c3723f68adf1210" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 44465 } }
I20260620 20:16:44.881830 24860 catalog_manager.cc:5495] T 485abc1387924f4aa537abe9bd114c9f P 5a07c01f0567489fb51b6bb676a3764f reported cstate change: term changed from 0 to 1, leader changed from <none> to 5a07c01f0567489fb51b6bb676a3764f (127.16.217.1). New cstate: current_term: 1 leader_uuid: "5a07c01f0567489fb51b6bb676a3764f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a07c01f0567489fb51b6bb676a3764f" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 43535 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a64d4977929340b7a5de6108995a78fa" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 36825 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "386309b3999d45769c3723f68adf1210" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 44465 } health_report { overall_health: UNKNOWN } } }
I20260620 20:16:44.915014 25370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:44.908318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38034 (local address 127.16.217.1:43535)
0620 20:16:44.908518 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0620 20:16:44.908528 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 20:16:44.908552 (+ 24us) server_negotiation.cc:407] Connection header received
0620 20:16:44.908697 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:16:44.908703 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:16:44.908822 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:16:44.908927 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 20:16:44.909370 (+ 443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:44.910680 (+ 1310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:44.912650 (+ 1970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:44.913487 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:44.913617 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:44.914162 (+ 545us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 20:16:44.914328 (+ 166us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 20:16:44.914429 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0620 20:16:44.914605 (+ 176us) server_negotiation.cc:299] Negotiation successful
0620 20:16:44.914687 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 20:16:44.918269 25157 raft_consensus.cc:1261] T 485abc1387924f4aa537abe9bd114c9f P a64d4977929340b7a5de6108995a78fa [term 1 FOLLOWER]: Refusing update from remote peer 5a07c01f0567489fb51b6bb676a3764f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:16:44.918318 25301 raft_consensus.cc:1261] T 485abc1387924f4aa537abe9bd114c9f P 386309b3999d45769c3723f68adf1210 [term 1 FOLLOWER]: Refusing update from remote peer 5a07c01f0567489fb51b6bb676a3764f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:16:44.918885 25386 consensus_queue.cc:1035] T 485abc1387924f4aa537abe9bd114c9f P 5a07c01f0567489fb51b6bb676a3764f [LEADER]: Connected to new peer: Peer: permanent_uuid: "a64d4977929340b7a5de6108995a78fa" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 36825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:16:44.919157 25381 consensus_queue.cc:1035] T 485abc1387924f4aa537abe9bd114c9f P 5a07c01f0567489fb51b6bb676a3764f [LEADER]: Connected to new peer: Peer: permanent_uuid: "386309b3999d45769c3723f68adf1210" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 44465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:16:44.926820 25391 mvcc.cc:205] Tried to move back new op lower bound from 7299017133738909696 to 7299017133586587648. Current Snapshot: MvccSnapshot[applied={T|T < 7299017133738909696}]
I20260620 20:16:44.933683 24859 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40594:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 20:16:44.934054 24859 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40594:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 20:16:44.937588 24859 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 827ffa1470674fdfa9e771771e6a7d94: Sending DeleteTablet for 3 replicas of tablet 485abc1387924f4aa537abe9bd114c9f
I20260620 20:16:44.938472 24998 tablet_service.cc:1514] Processing DeleteTablet for tablet 485abc1387924f4aa537abe9bd114c9f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:16:44 UTC) from {username='oryx', principal='oryx/127.16.217.62@KRBTEST.COM'} at 127.0.0.1:38004
I20260620 20:16:44.938993 25137 tablet_service.cc:1514] Processing DeleteTablet for tablet 485abc1387924f4aa537abe9bd114c9f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:16:44 UTC) from {username='oryx', principal='oryx/127.16.217.62@KRBTEST.COM'} at 127.0.0.1:38734
I20260620 20:16:44.939096 25400 tablet_replica.cc:330] T 485abc1387924f4aa537abe9bd114c9f P 5a07c01f0567489fb51b6bb676a3764f: stopping tablet replica
I20260620 20:16:44.939106 25138 tablet_service.cc:1514] Processing DeleteTablet for tablet 485abc1387924f4aa537abe9bd114c9f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:16:44 UTC) from {username='oryx', principal='oryx/127.16.217.62@KRBTEST.COM'} at 127.0.0.1:38734
I20260620 20:16:44.939306 25400 raft_consensus.cc:2229] T 485abc1387924f4aa537abe9bd114c9f P 5a07c01f0567489fb51b6bb676a3764f [term 1 LEADER]: Raft consensus shutting down.
I20260620 20:16:44.939368 25399 tablet_replica.cc:330] T 485abc1387924f4aa537abe9bd114c9f P a64d4977929340b7a5de6108995a78fa: stopping tablet replica
I20260620 20:16:44.939512 25399 raft_consensus.cc:2229] T 485abc1387924f4aa537abe9bd114c9f P a64d4977929340b7a5de6108995a78fa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 20:16:44.939661 25399 pending_rounds.cc:62] T 485abc1387924f4aa537abe9bd114c9f P a64d4977929340b7a5de6108995a78fa: Trying to abort 2 pending ops.
I20260620 20:16:44.939700 25400 raft_consensus.cc:2258] T 485abc1387924f4aa537abe9bd114c9f P 5a07c01f0567489fb51b6bb676a3764f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:16:44.939991 25278 tablet_service.cc:1514] Processing DeleteTablet for tablet 485abc1387924f4aa537abe9bd114c9f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:16:44 UTC) from {username='oryx', principal='oryx/127.16.217.62@KRBTEST.COM'} at 127.0.0.1:49776
I20260620 20:16:44.939774 25399 pending_rounds.cc:68] T 485abc1387924f4aa537abe9bd114c9f P a64d4977929340b7a5de6108995a78fa: Aborting op as it isn't in flight: id { term: 1 index: 2 } timestamp: 7299017133738909696 op_type: WRITE_OP write_request { tablet_id: "485abc1387924f4aa537abe9bd114c9f" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "val" type: INT32 is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\003\00090\000\0001\324\000\000" indirect_data: "<redacted>""" } external_consistency_mode: CLIENT_PROPAGATED authz_token { token_data: "\010\330\354\333\321\006\"8\n\ntest-admin\022*\n e2de26cd60424214be14e198b28ce757\020\001\030\001 \001(\001" signature: "<redacted>""\211X)tA\204[\252j2\314\275\003\264G\277\246N\347ge\303\024\353\013\023\312<^\027^\270\255\373f-\257\3571\216\350\277l\274z\261%(\nZ\335z,\255\333]l\375\221\264M\361\211\306" signing_key_seq_num: 0 } } request_id { client_id: "0a5faf7f2dd8431385e89a4cbc499752" seq_no: 0 first_incomplete_seq_no: 0 attempt_no: 0 }
I20260620 20:16:44.940460 25401 tablet_replica.cc:330] T 485abc1387924f4aa537abe9bd114c9f P 386309b3999d45769c3723f68adf1210: stopping tablet replica
I20260620 20:16:44.940460 25399 pending_rounds.cc:68] T 485abc1387924f4aa537abe9bd114c9f P a64d4977929340b7a5de6108995a78fa: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7299017133586587648 op_type: NO_OP noop_request { }
I20260620 20:16:44.940583 25399 raft_consensus.cc:2866] T 485abc1387924f4aa537abe9bd114c9f P a64d4977929340b7a5de6108995a78fa [term 1 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20260620 20:16:44.940639 25401 raft_consensus.cc:2229] T 485abc1387924f4aa537abe9bd114c9f P 386309b3999d45769c3723f68adf1210 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 20:16:44.940732 25399 raft_consensus.cc:2258] T 485abc1387924f4aa537abe9bd114c9f P a64d4977929340b7a5de6108995a78fa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:16:44.940861 25401 raft_consensus.cc:2258] T 485abc1387924f4aa537abe9bd114c9f P 386309b3999d45769c3723f68adf1210 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:16:44.941743 24860 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40594:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260620 20:16:44.942425 25400 ts_tablet_manager.cc:1905] T 485abc1387924f4aa537abe9bd114c9f P 5a07c01f0567489fb51b6bb676a3764f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:16:44.942473 25401 ts_tablet_manager.cc:1905] T 485abc1387924f4aa537abe9bd114c9f P 386309b3999d45769c3723f68adf1210: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260620 20:16:44.942472 24860 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 20:16:44.948482 25399 ts_tablet_manager.cc:1905] T 485abc1387924f4aa537abe9bd114c9f P a64d4977929340b7a5de6108995a78fa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:16:44.949098 25401 ts_tablet_manager.cc:1918] T 485abc1387924f4aa537abe9bd114c9f P 386309b3999d45769c3723f68adf1210: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 20:16:44.949218 25401 log.cc:1196] T 485abc1387924f4aa537abe9bd114c9f P 386309b3999d45769c3723f68adf1210: Deleting WAL directory at /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986536687152-17252-0/minicluster-data/ts-2/wal/wals/485abc1387924f4aa537abe9bd114c9f
I20260620 20:16:44.949594 25401 ts_tablet_manager.cc:1939] T 485abc1387924f4aa537abe9bd114c9f P 386309b3999d45769c3723f68adf1210: Deleting consensus metadata
I20260620 20:16:44.950711 24843 catalog_manager.cc:4841] TS 386309b3999d45769c3723f68adf1210 (127.16.217.3:44465): tablet 485abc1387924f4aa537abe9bd114c9f (table test-table [id=e2de26cd60424214be14e198b28ce757]) successfully deleted
I20260620 20:16:44.955044 24998 tablet_service.cc:1467] Processing CreateTablet for tablet 722c4188b4504e38a8e671cf3fde4fd4 (DEFAULT_TABLE table=test-table [id=1337056ce47e4792acfaa8692a52e18c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:16:44.955130 25138 tablet_service.cc:1467] Processing CreateTablet for tablet 722c4188b4504e38a8e671cf3fde4fd4 (DEFAULT_TABLE table=test-table [id=1337056ce47e4792acfaa8692a52e18c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:16:44.955507 24998 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 722c4188b4504e38a8e671cf3fde4fd4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:16:44.955533 25138 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 722c4188b4504e38a8e671cf3fde4fd4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:16:44.958065 25399 ts_tablet_manager.cc:1918] T 485abc1387924f4aa537abe9bd114c9f P a64d4977929340b7a5de6108995a78fa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 20:16:44.958199 25399 log.cc:1196] T 485abc1387924f4aa537abe9bd114c9f P a64d4977929340b7a5de6108995a78fa: Deleting WAL directory at /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986536687152-17252-0/minicluster-data/ts-1/wal/wals/485abc1387924f4aa537abe9bd114c9f
I20260620 20:16:44.958479 25278 tablet_service.cc:1467] Processing CreateTablet for tablet 722c4188b4504e38a8e671cf3fde4fd4 (DEFAULT_TABLE table=test-table [id=1337056ce47e4792acfaa8692a52e18c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:16:44.958640 25399 ts_tablet_manager.cc:1939] T 485abc1387924f4aa537abe9bd114c9f P a64d4977929340b7a5de6108995a78fa: Deleting consensus metadata
I20260620 20:16:44.958891 25278 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 722c4188b4504e38a8e671cf3fde4fd4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:16:44.959704 24843 catalog_manager.cc:4841] TS a64d4977929340b7a5de6108995a78fa (127.16.217.2:36825): tablet 485abc1387924f4aa537abe9bd114c9f (table test-table [id=e2de26cd60424214be14e198b28ce757]) successfully deleted
W20260620 20:16:44.960532 24843 catalog_manager.cc:4808] TS a64d4977929340b7a5de6108995a78fa (127.16.217.2:36825): delete failed for tablet 485abc1387924f4aa537abe9bd114c9f because the tablet was not found. No further retry: Not found: Tablet not found: 485abc1387924f4aa537abe9bd114c9f
I20260620 20:16:44.961481 25374 tablet_bootstrap.cc:492] T 722c4188b4504e38a8e671cf3fde4fd4 P a64d4977929340b7a5de6108995a78fa: Bootstrap starting.
I20260620 20:16:44.963001 25374 tablet_bootstrap.cc:654] T 722c4188b4504e38a8e671cf3fde4fd4 P a64d4977929340b7a5de6108995a78fa: Neither blocks nor log segments found. Creating new log.
I20260620 20:16:44.963436 25400 ts_tablet_manager.cc:1918] T 485abc1387924f4aa537abe9bd114c9f P 5a07c01f0567489fb51b6bb676a3764f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 20:16:44.963555 25400 log.cc:1196] T 485abc1387924f4aa537abe9bd114c9f P 5a07c01f0567489fb51b6bb676a3764f: Deleting WAL directory at /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986536687152-17252-0/minicluster-data/ts-0/wal/wals/485abc1387924f4aa537abe9bd114c9f
I20260620 20:16:44.963961 25400 ts_tablet_manager.cc:1939] T 485abc1387924f4aa537abe9bd114c9f P 5a07c01f0567489fb51b6bb676a3764f: Deleting consensus metadata
I20260620 20:16:44.964807 25374 tablet_bootstrap.cc:492] T 722c4188b4504e38a8e671cf3fde4fd4 P a64d4977929340b7a5de6108995a78fa: No bootstrap required, opened a new log
I20260620 20:16:44.964951 25374 ts_tablet_manager.cc:1397] T 722c4188b4504e38a8e671cf3fde4fd4 P a64d4977929340b7a5de6108995a78fa: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260620 20:16:44.965073 24846 catalog_manager.cc:4841] TS 5a07c01f0567489fb51b6bb676a3764f (127.16.217.1:43535): tablet 485abc1387924f4aa537abe9bd114c9f (table test-table [id=e2de26cd60424214be14e198b28ce757]) successfully deleted
I20260620 20:16:44.965286 25375 tablet_bootstrap.cc:492] T 722c4188b4504e38a8e671cf3fde4fd4 P 386309b3999d45769c3723f68adf1210: Bootstrap starting.
I20260620 20:16:44.965703 25374 raft_consensus.cc:348] T 722c4188b4504e38a8e671cf3fde4fd4 P a64d4977929340b7a5de6108995a78fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a07c01f0567489fb51b6bb676a3764f" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 43535 } } peers { permanent_uuid: "a64d4977929340b7a5de6108995a78fa" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 36825 } } peers { permanent_uuid: "386309b3999d45769c3723f68adf1210" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 44465 } }
I20260620 20:16:44.965955 25374 raft_consensus.cc:374] T 722c4188b4504e38a8e671cf3fde4fd4 P a64d4977929340b7a5de6108995a78fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:16:44.966065 25374 raft_consensus.cc:729] T 722c4188b4504e38a8e671cf3fde4fd4 P a64d4977929340b7a5de6108995a78fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64d4977929340b7a5de6108995a78fa, State: Initialized, Role: FOLLOWER
I20260620 20:16:44.966306 25374 consensus_queue.cc:260] T 722c4188b4504e38a8e671cf3fde4fd4 P a64d4977929340b7a5de6108995a78fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a07c01f0567489fb51b6bb676a3764f" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 43535 } } peers { permanent_uuid: "a64d4977929340b7a5de6108995a78fa" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 36825 } } peers { permanent_uuid: "386309b3999d45769c3723f68adf1210" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 44465 } }
I20260620 20:16:44.966907 25375 tablet_bootstrap.cc:654] T 722c4188b4504e38a8e671cf3fde4fd4 P 386309b3999d45769c3723f68adf1210: Neither blocks nor log segments found. Creating new log.
I20260620 20:16:44.967263 25374 ts_tablet_manager.cc:1428] T 722c4188b4504e38a8e671cf3fde4fd4 P a64d4977929340b7a5de6108995a78fa: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 20:16:44.967321 25373 tablet_bootstrap.cc:492] T 722c4188b4504e38a8e671cf3fde4fd4 P 5a07c01f0567489fb51b6bb676a3764f: Bootstrap starting.
I20260620 20:16:44.968924 25373 tablet_bootstrap.cc:654] T 722c4188b4504e38a8e671cf3fde4fd4 P 5a07c01f0567489fb51b6bb676a3764f: Neither blocks nor log segments found. Creating new log.
I20260620 20:16:44.969164 25375 tablet_bootstrap.cc:492] T 722c4188b4504e38a8e671cf3fde4fd4 P 386309b3999d45769c3723f68adf1210: No bootstrap required, opened a new log
I20260620 20:16:44.969331 25375 ts_tablet_manager.cc:1397] T 722c4188b4504e38a8e671cf3fde4fd4 P 386309b3999d45769c3723f68adf1210: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260620 20:16:44.971302 25373 tablet_bootstrap.cc:492] T 722c4188b4504e38a8e671cf3fde4fd4 P 5a07c01f0567489fb51b6bb676a3764f: No bootstrap required, opened a new log
I20260620 20:16:44.971491 25373 ts_tablet_manager.cc:1397] T 722c4188b4504e38a8e671cf3fde4fd4 P 5a07c01f0567489fb51b6bb676a3764f: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20260620 20:16:44.971345 25375 raft_consensus.cc:348] T 722c4188b4504e38a8e671cf3fde4fd4 P 386309b3999d45769c3723f68adf1210 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a07c01f0567489fb51b6bb676a3764f" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 43535 } } peers { permanent_uuid: "a64d4977929340b7a5de6108995a78fa" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 36825 } } peers { permanent_uuid: "386309b3999d45769c3723f68adf1210" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 44465 } }
I20260620 20:16:44.971588 25375 raft_consensus.cc:374] T 722c4188b4504e38a8e671cf3fde4fd4 P 386309b3999d45769c3723f68adf1210 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:16:44.971675 25375 raft_consensus.cc:729] T 722c4188b4504e38a8e671cf3fde4fd4 P 386309b3999d45769c3723f68adf1210 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 386309b3999d45769c3723f68adf1210, State: Initialized, Role: FOLLOWER
I20260620 20:16:44.971953 25375 consensus_queue.cc:260] T 722c4188b4504e38a8e671cf3fde4fd4 P 386309b3999d45769c3723f68adf1210 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a07c01f0567489fb51b6bb676a3764f" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 43535 } } peers { permanent_uuid: "a64d4977929340b7a5de6108995a78fa" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 36825 } } peers { permanent_uuid: "386309b3999d45769c3723f68adf1210" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 44465 } }
I20260620 20:16:44.972275 25373 raft_consensus.cc:348] T 722c4188b4504e38a8e671cf3fde4fd4 P 5a07c01f0567489fb51b6bb676a3764f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a07c01f0567489fb51b6bb676a3764f" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 43535 } } peers { permanent_uuid: "a64d4977929340b7a5de6108995a78fa" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 36825 } } peers { permanent_uuid: "386309b3999d45769c3723f68adf1210" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 44465 } }
I20260620 20:16:44.972534 25373 raft_consensus.cc:374] T 722c4188b4504e38a8e671cf3fde4fd4 P 5a07c01f0567489fb51b6bb676a3764f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:16:44.972611 25373 raft_consensus.cc:729] T 722c4188b4504e38a8e671cf3fde4fd4 P 5a07c01f0567489fb51b6bb676a3764f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a07c01f0567489fb51b6bb676a3764f, State: Initialized, Role: FOLLOWER
I20260620 20:16:44.972834 25373 consensus_queue.cc:260] T 722c4188b4504e38a8e671cf3fde4fd4 P 5a07c01f0567489fb51b6bb676a3764f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a07c01f0567489fb51b6bb676a3764f" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 43535 } } peers { permanent_uuid: "a64d4977929340b7a5de6108995a78fa" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 36825 } } peers { permanent_uuid: "386309b3999d45769c3723f68adf1210" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 44465 } }
I20260620 20:16:44.973664 25375 ts_tablet_manager.cc:1428] T 722c4188b4504e38a8e671cf3fde4fd4 P 386309b3999d45769c3723f68adf1210: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20260620 20:16:44.974936 25373 ts_tablet_manager.cc:1428] T 722c4188b4504e38a8e671cf3fde4fd4 P 5a07c01f0567489fb51b6bb676a3764f: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260620 20:16:45.053354 25380 raft_consensus.cc:482] T 722c4188b4504e38a8e671cf3fde4fd4 P 386309b3999d45769c3723f68adf1210 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 20:16:45.053592 25380 raft_consensus.cc:504] T 722c4188b4504e38a8e671cf3fde4fd4 P 386309b3999d45769c3723f68adf1210 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a07c01f0567489fb51b6bb676a3764f" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 43535 } } peers { permanent_uuid: "a64d4977929340b7a5de6108995a78fa" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 36825 } } peers { permanent_uuid: "386309b3999d45769c3723f68adf1210" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 44465 } }
I20260620 20:16:45.055261 25380 leader_election.cc:288] T 722c4188b4504e38a8e671cf3fde4fd4 P 386309b3999d45769c3723f68adf1210 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5a07c01f0567489fb51b6bb676a3764f (127.16.217.1:43535), a64d4977929340b7a5de6108995a78fa (127.16.217.2:36825)
I20260620 20:16:45.069607 25370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:45.055770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.217.3:39531 (local address 127.16.217.1:43535)
0620 20:16:45.055955 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0620 20:16:45.055962 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:16:45.062449 (+ 6487us) server_negotiation.cc:407] Connection header received
0620 20:16:45.062753 (+ 304us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:16:45.062758 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:16:45.062832 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:16:45.062900 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:16:45.063714 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:45.064747 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:45.067156 (+ 2409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:45.068361 (+ 1205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:45.068488 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:45.068559 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 20:16:45.069151 (+ 592us) server_negotiation.cc:299] Negotiation successful
0620 20:16:45.069334 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 20:16:45.070345 25450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:45.055433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.1:43535 (local address 127.16.217.3:39531)
0620 20:16:45.062339 (+ 6906us) negotiation.cc:107] Waiting for socket to connect
0620 20:16:45.062360 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 20:16:45.062562 (+ 202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:16:45.063117 (+ 555us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:16:45.063121 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:16:45.063152 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:16:45.063524 (+ 372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:16:45.063535 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:16:45.064922 (+ 1387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:16:45.064928 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:16:45.066999 (+ 2071us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:16:45.067013 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:16:45.068758 (+ 1745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:16:45.068764 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:16:45.068971 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:45.069004 (+ 33us) client_negotiation.cc:769] Sending connection context
0620 20:16:45.069792 (+ 788us) client_negotiation.cc:240] Negotiation successful
0620 20:16:45.069962 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6750,"thread_start_us":14632,"threads_started":1}
I20260620 20:16:45.071174 25018 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "722c4188b4504e38a8e671cf3fde4fd4" candidate_uuid: "386309b3999d45769c3723f68adf1210" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a07c01f0567489fb51b6bb676a3764f" is_pre_election: true
I20260620 20:16:45.071463 25018 raft_consensus.cc:2454] T 722c4188b4504e38a8e671cf3fde4fd4 P 5a07c01f0567489fb51b6bb676a3764f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 386309b3999d45769c3723f68adf1210 in term 0.
I20260620 20:16:45.072021 25232 leader_election.cc:302] T 722c4188b4504e38a8e671cf3fde4fd4 P 386309b3999d45769c3723f68adf1210 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 386309b3999d45769c3723f68adf1210, 5a07c01f0567489fb51b6bb676a3764f; no voters:
I20260620 20:16:45.072692 25380 raft_consensus.cc:2781] T 722c4188b4504e38a8e671cf3fde4fd4 P 386309b3999d45769c3723f68adf1210 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 20:16:45.072826 25380 raft_consensus.cc:482] T 722c4188b4504e38a8e671cf3fde4fd4 P 386309b3999d45769c3723f68adf1210 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 20:16:45.072919 25380 raft_consensus.cc:3037] T 722c4188b4504e38a8e671cf3fde4fd4 P 386309b3999d45769c3723f68adf1210 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:16:45.076202 25380 raft_consensus.cc:504] T 722c4188b4504e38a8e671cf3fde4fd4 P 386309b3999d45769c3723f68adf1210 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a07c01f0567489fb51b6bb676a3764f" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 43535 } } peers { permanent_uuid: "a64d4977929340b7a5de6108995a78fa" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 36825 } } peers { permanent_uuid: "386309b3999d45769c3723f68adf1210" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 44465 } }
I20260620 20:16:45.076749 25451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:45.056012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.217.3:37133 (local address 127.16.217.2:36825)
0620 20:16:45.071019 (+ 15007us) server_negotiation.cc:206] Beginning negotiation
0620 20:16:45.071026 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:16:45.071045 (+ 19us) server_negotiation.cc:407] Connection header received
0620 20:16:45.071112 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:16:45.071117 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:16:45.071203 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:16:45.071321 (+ 118us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:16:45.071959 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:45.073076 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:45.074891 (+ 1815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:45.075715 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:45.075914 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:45.075985 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 20:16:45.076267 (+ 282us) server_negotiation.cc:299] Negotiation successful
0620 20:16:45.076402 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14848,"thread_start_us":14599,"threads_started":1}
I20260620 20:16:45.076858 25380 leader_election.cc:288] T 722c4188b4504e38a8e671cf3fde4fd4 P 386309b3999d45769c3723f68adf1210 [CANDIDATE]: Term 1 election: Requested vote from peers 5a07c01f0567489fb51b6bb676a3764f (127.16.217.1:43535), a64d4977929340b7a5de6108995a78fa (127.16.217.2:36825)
I20260620 20:16:45.077298 25449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:45.055673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.2:36825 (local address 127.16.217.3:37133)
0620 20:16:45.070408 (+ 14735us) negotiation.cc:107] Waiting for socket to connect
0620 20:16:45.070429 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 20:16:45.070544 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:16:45.071362 (+ 818us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:16:45.071368 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:16:45.071394 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:16:45.071695 (+ 301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:16:45.071705 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:16:45.073202 (+ 1497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:16:45.073210 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:16:45.074720 (+ 1510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:16:45.074731 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:16:45.075932 (+ 1201us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:16:45.075938 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:16:45.076094 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:45.076116 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 20:16:45.076235 (+ 119us) client_negotiation.cc:240] Negotiation successful
0620 20:16:45.076371 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14586,"thread_start_us":6370,"threads_started":1}
I20260620 20:16:45.077620 25018 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "722c4188b4504e38a8e671cf3fde4fd4" candidate_uuid: "386309b3999d45769c3723f68adf1210" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a07c01f0567489fb51b6bb676a3764f"
I20260620 20:16:45.077847 25018 raft_consensus.cc:3037] T 722c4188b4504e38a8e671cf3fde4fd4 P 5a07c01f0567489fb51b6bb676a3764f [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:16:45.078071 25158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "722c4188b4504e38a8e671cf3fde4fd4" candidate_uuid: "386309b3999d45769c3723f68adf1210" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64d4977929340b7a5de6108995a78fa"
I20260620 20:16:45.078275 25158 raft_consensus.cc:3037] T 722c4188b4504e38a8e671cf3fde4fd4 P a64d4977929340b7a5de6108995a78fa [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:16:45.080202 25018 raft_consensus.cc:2454] T 722c4188b4504e38a8e671cf3fde4fd4 P 5a07c01f0567489fb51b6bb676a3764f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 386309b3999d45769c3723f68adf1210 in term 1.
I20260620 20:16:45.080436 25158 raft_consensus.cc:2454] T 722c4188b4504e38a8e671cf3fde4fd4 P a64d4977929340b7a5de6108995a78fa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 386309b3999d45769c3723f68adf1210 in term 1.
I20260620 20:16:45.080607 25232 leader_election.cc:302] T 722c4188b4504e38a8e671cf3fde4fd4 P 386309b3999d45769c3723f68adf1210 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 386309b3999d45769c3723f68adf1210, 5a07c01f0567489fb51b6bb676a3764f; no voters:
I20260620 20:16:45.080600 25157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "722c4188b4504e38a8e671cf3fde4fd4" candidate_uuid: "386309b3999d45769c3723f68adf1210" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64d4977929340b7a5de6108995a78fa" is_pre_election: true
I20260620 20:16:45.080850 25157 raft_consensus.cc:2362] T 722c4188b4504e38a8e671cf3fde4fd4 P a64d4977929340b7a5de6108995a78fa [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 386309b3999d45769c3723f68adf1210 in term 1. Re-sending same reply.
I20260620 20:16:45.080899 25380 raft_consensus.cc:2781] T 722c4188b4504e38a8e671cf3fde4fd4 P 386309b3999d45769c3723f68adf1210 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:16:45.081338 25380 raft_consensus.cc:686] T 722c4188b4504e38a8e671cf3fde4fd4 P 386309b3999d45769c3723f68adf1210 [term 1 LEADER]: Becoming Leader. State: Replica: 386309b3999d45769c3723f68adf1210, State: Running, Role: LEADER
I20260620 20:16:45.081743 25380 consensus_queue.cc:237] T 722c4188b4504e38a8e671cf3fde4fd4 P 386309b3999d45769c3723f68adf1210 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a07c01f0567489fb51b6bb676a3764f" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 43535 } } peers { permanent_uuid: "a64d4977929340b7a5de6108995a78fa" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 36825 } } peers { permanent_uuid: "386309b3999d45769c3723f68adf1210" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 44465 } }
I20260620 20:16:45.085484 24860 catalog_manager.cc:5495] T 722c4188b4504e38a8e671cf3fde4fd4 P 386309b3999d45769c3723f68adf1210 reported cstate change: term changed from 0 to 1, leader changed from <none> to 386309b3999d45769c3723f68adf1210 (127.16.217.3). New cstate: current_term: 1 leader_uuid: "386309b3999d45769c3723f68adf1210" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a07c01f0567489fb51b6bb676a3764f" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 43535 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a64d4977929340b7a5de6108995a78fa" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 36825 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "386309b3999d45769c3723f68adf1210" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 44465 } health_report { overall_health: HEALTHY } } }
I20260620 20:16:45.107024 25455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:45.101520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49784 (local address 127.16.217.3:44465)
0620 20:16:45.102089 (+ 569us) server_negotiation.cc:206] Beginning negotiation
0620 20:16:45.102093 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:16:45.102111 (+ 18us) server_negotiation.cc:407] Connection header received
0620 20:16:45.102185 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:16:45.102188 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:16:45.102258 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:16:45.102391 (+ 133us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 20:16:45.102729 (+ 338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:45.103490 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:45.104739 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:45.105116 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:45.105204 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:45.105633 (+ 429us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 20:16:45.106318 (+ 685us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 20:16:45.106426 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0620 20:16:45.106575 (+ 149us) server_negotiation.cc:299] Negotiation successful
0620 20:16:45.106650 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":476,"thread_start_us":307,"threads_started":1}
I20260620 20:16:45.121227 25370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:45.116482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.217.3:54689 (local address 127.16.217.1:43535)
0620 20:16:45.116666 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0620 20:16:45.116673 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:16:45.116870 (+ 197us) server_negotiation.cc:407] Connection header received
0620 20:16:45.117073 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:16:45.117077 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:16:45.117145 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:16:45.117245 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 20:16:45.117662 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:45.118469 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:45.119766 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:45.120199 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:45.120306 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:45.120658 (+ 352us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 20:16:45.120756 (+ 98us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 20:16:45.120828 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 20:16:45.120954 (+ 126us) server_negotiation.cc:299] Negotiation successful
0620 20:16:45.121014 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 20:16:45.121287 25458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:45.116158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.1:43535 (local address 127.16.217.3:54689)
0620 20:16:45.116780 (+ 622us) negotiation.cc:107] Waiting for socket to connect
0620 20:16:45.116798 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 20:16:45.116960 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:16:45.117266 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:16:45.117269 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:16:45.117279 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 20:16:45.117535 (+ 256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:16:45.117541 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:16:45.118587 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:16:45.118590 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:16:45.119635 (+ 1045us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:16:45.119645 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:16:45.120351 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:16:45.120355 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:16:45.120464 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:45.120509 (+ 45us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 20:16:45.120842 (+ 333us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 20:16:45.120856 (+ 14us) client_negotiation.cc:769] Sending connection context
0620 20:16:45.120916 (+ 60us) client_negotiation.cc:240] Negotiation successful
0620 20:16:45.121006 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":481,"thread_start_us":322,"threads_started":1}
I20260620 20:16:45.128955 25157 raft_consensus.cc:1261] T 722c4188b4504e38a8e671cf3fde4fd4 P a64d4977929340b7a5de6108995a78fa [term 1 FOLLOWER]: Refusing update from remote peer 386309b3999d45769c3723f68adf1210: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:16:45.129256 25018 raft_consensus.cc:1261] T 722c4188b4504e38a8e671cf3fde4fd4 P 5a07c01f0567489fb51b6bb676a3764f [term 1 FOLLOWER]: Refusing update from remote peer 386309b3999d45769c3723f68adf1210: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:16:45.129570 25452 consensus_queue.cc:1035] T 722c4188b4504e38a8e671cf3fde4fd4 P 386309b3999d45769c3723f68adf1210 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a64d4977929340b7a5de6108995a78fa" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 36825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:16:45.129948 25380 consensus_queue.cc:1035] T 722c4188b4504e38a8e671cf3fde4fd4 P 386309b3999d45769c3723f68adf1210 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a07c01f0567489fb51b6bb676a3764f" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 43535 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:16:45.134717 25390 mvcc.cc:205] Tried to move back new op lower bound from 7299017134603374592 to 7299017134419070976. Current Snapshot: MvccSnapshot[applied={T|T < 7299017134603374592}]
I20260620 20:16:45.153714 25455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:45.148031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.217.1:54109 (local address 127.16.217.3:44465)
0620 20:16:45.148217 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0620 20:16:45.148224 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:16:45.148494 (+ 270us) server_negotiation.cc:407] Connection header received
0620 20:16:45.148730 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:16:45.148737 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:16:45.148880 (+ 143us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:16:45.148970 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 20:16:45.149650 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:45.150443 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:45.152064 (+ 1621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:45.152694 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:45.152803 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:45.153134 (+ 331us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 20:16:45.153249 (+ 115us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 20:16:45.153337 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0620 20:16:45.153432 (+ 95us) server_negotiation.cc:299] Negotiation successful
0620 20:16:45.153489 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 20:16:45.153759 25462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:45.147942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.3:44465 (local address 127.16.217.1:54109)
0620 20:16:45.148396 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0620 20:16:45.148412 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 20:16:45.148539 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:16:45.149025 (+ 486us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:16:45.149070 (+ 45us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:16:45.149107 (+ 37us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 20:16:45.149491 (+ 384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:16:45.149503 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:16:45.150551 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:16:45.150555 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:16:45.151936 (+ 1381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:16:45.151948 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:16:45.152825 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:16:45.152830 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:16:45.152957 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:45.152991 (+ 34us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 20:16:45.153344 (+ 353us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 20:16:45.153353 (+ 9us) client_negotiation.cc:769] Sending connection context
0620 20:16:45.153410 (+ 57us) client_negotiation.cc:240] Negotiation successful
0620 20:16:45.153507 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":285,"threads_started":1}
I20260620 20:16:45.184810 24859 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40594:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 20:16:45.185009 24859 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40594:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 20:16:45.188247 24859 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 827ffa1470674fdfa9e771771e6a7d94: Sending DeleteTablet for 3 replicas of tablet 722c4188b4504e38a8e671cf3fde4fd4
I20260620 20:16:45.189406 24998 tablet_service.cc:1514] Processing DeleteTablet for tablet 722c4188b4504e38a8e671cf3fde4fd4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:16:45 UTC) from {username='oryx', principal='oryx/127.16.217.62@KRBTEST.COM'} at 127.0.0.1:38004
I20260620 20:16:45.189538 25138 tablet_service.cc:1514] Processing DeleteTablet for tablet 722c4188b4504e38a8e671cf3fde4fd4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:16:45 UTC) from {username='oryx', principal='oryx/127.16.217.62@KRBTEST.COM'} at 127.0.0.1:38734
I20260620 20:16:45.189747 25400 tablet_replica.cc:330] T 722c4188b4504e38a8e671cf3fde4fd4 P 5a07c01f0567489fb51b6bb676a3764f: stopping tablet replica
I20260620 20:16:45.190013 25400 raft_consensus.cc:2229] T 722c4188b4504e38a8e671cf3fde4fd4 P 5a07c01f0567489fb51b6bb676a3764f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 20:16:45.190088 25399 tablet_replica.cc:330] T 722c4188b4504e38a8e671cf3fde4fd4 P a64d4977929340b7a5de6108995a78fa: stopping tablet replica
I20260620 20:16:45.190244 25400 raft_consensus.cc:2258] T 722c4188b4504e38a8e671cf3fde4fd4 P 5a07c01f0567489fb51b6bb676a3764f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:16:45.190498 25399 raft_consensus.cc:2229] T 722c4188b4504e38a8e671cf3fde4fd4 P a64d4977929340b7a5de6108995a78fa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 20:16:45.190838 25399 raft_consensus.cc:2258] T 722c4188b4504e38a8e671cf3fde4fd4 P a64d4977929340b7a5de6108995a78fa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:16:45.191606 25400 ts_tablet_manager.cc:1905] T 722c4188b4504e38a8e671cf3fde4fd4 P 5a07c01f0567489fb51b6bb676a3764f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:16:45.195137 17252 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu pid 24931
I20260620 20:16:45.196553 25399 ts_tablet_manager.cc:1905] T 722c4188b4504e38a8e671cf3fde4fd4 P a64d4977929340b7a5de6108995a78fa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:16:45.197605 25278 tablet_service.cc:1514] Processing DeleteTablet for tablet 722c4188b4504e38a8e671cf3fde4fd4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:16:45 UTC) from {username='oryx', principal='oryx/127.16.217.62@KRBTEST.COM'} at 127.0.0.1:49776
I20260620 20:16:45.197913 25401 tablet_replica.cc:330] T 722c4188b4504e38a8e671cf3fde4fd4 P 386309b3999d45769c3723f68adf1210: stopping tablet replica
I20260620 20:16:45.198105 25401 raft_consensus.cc:2229] T 722c4188b4504e38a8e671cf3fde4fd4 P 386309b3999d45769c3723f68adf1210 [term 1 LEADER]: Raft consensus shutting down.
I20260620 20:16:45.198597 25401 raft_consensus.cc:2258] T 722c4188b4504e38a8e671cf3fde4fd4 P 386309b3999d45769c3723f68adf1210 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:16:45.198887 25400 ts_tablet_manager.cc:1918] T 722c4188b4504e38a8e671cf3fde4fd4 P 5a07c01f0567489fb51b6bb676a3764f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 20:16:45.199133 25400 log.cc:1196] T 722c4188b4504e38a8e671cf3fde4fd4 P 5a07c01f0567489fb51b6bb676a3764f: Deleting WAL directory at /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986536687152-17252-0/minicluster-data/ts-0/wal/wals/722c4188b4504e38a8e671cf3fde4fd4
I20260620 20:16:45.199644 25400 ts_tablet_manager.cc:1939] T 722c4188b4504e38a8e671cf3fde4fd4 P 5a07c01f0567489fb51b6bb676a3764f: Deleting consensus metadata
I20260620 20:16:45.199738 25401 ts_tablet_manager.cc:1905] T 722c4188b4504e38a8e671cf3fde4fd4 P 386309b3999d45769c3723f68adf1210: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:16:45.200845 24846 catalog_manager.cc:4841] TS 5a07c01f0567489fb51b6bb676a3764f (127.16.217.1:43535): tablet 722c4188b4504e38a8e671cf3fde4fd4 (table test-table [id=1337056ce47e4792acfaa8692a52e18c]) successfully deleted
I20260620 20:16:45.204221 25399 ts_tablet_manager.cc:1918] T 722c4188b4504e38a8e671cf3fde4fd4 P a64d4977929340b7a5de6108995a78fa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 20:16:45.204419 25399 log.cc:1196] T 722c4188b4504e38a8e671cf3fde4fd4 P a64d4977929340b7a5de6108995a78fa: Deleting WAL directory at /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986536687152-17252-0/minicluster-data/ts-1/wal/wals/722c4188b4504e38a8e671cf3fde4fd4
I20260620 20:16:45.204873 25399 ts_tablet_manager.cc:1939] T 722c4188b4504e38a8e671cf3fde4fd4 P a64d4977929340b7a5de6108995a78fa: Deleting consensus metadata
I20260620 20:16:45.205946 24843 catalog_manager.cc:4841] TS a64d4977929340b7a5de6108995a78fa (127.16.217.2:36825): tablet 722c4188b4504e38a8e671cf3fde4fd4 (table test-table [id=1337056ce47e4792acfaa8692a52e18c]) successfully deleted
I20260620 20:16:45.206452 25401 ts_tablet_manager.cc:1918] T 722c4188b4504e38a8e671cf3fde4fd4 P 386309b3999d45769c3723f68adf1210: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 20:16:45.206598 25401 log.cc:1196] T 722c4188b4504e38a8e671cf3fde4fd4 P 386309b3999d45769c3723f68adf1210: Deleting WAL directory at /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986536687152-17252-0/minicluster-data/ts-2/wal/wals/722c4188b4504e38a8e671cf3fde4fd4
I20260620 20:16:45.207053 25401 ts_tablet_manager.cc:1939] T 722c4188b4504e38a8e671cf3fde4fd4 P 386309b3999d45769c3723f68adf1210: Deleting consensus metadata
I20260620 20:16:45.208243 24843 catalog_manager.cc:4841] TS 386309b3999d45769c3723f68adf1210 (127.16.217.3:44465): tablet 722c4188b4504e38a8e671cf3fde4fd4 (table test-table [id=1337056ce47e4792acfaa8692a52e18c]) successfully deleted
Jun 20 20:16:45 dist-test-slave-2hpg krb5kdc[24814](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986604, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.16.217.1@KRBTEST.COM
I20260620 20:16:45.223495 25370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:45.199373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38044 (local address 127.16.217.1:43535)
0620 20:16:45.199927 (+ 554us) server_negotiation.cc:206] Beginning negotiation
0620 20:16:45.199934 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:16:45.206750 (+ 6816us) server_negotiation.cc:407] Connection header received
0620 20:16:45.207055 (+ 305us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:16:45.207063 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:16:45.207155 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:16:45.207298 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:16:45.208791 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:45.209993 (+ 1202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:45.211224 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:45.211936 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:45.212087 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:45.215986 (+ 3899us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:16:45.216006 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:16:45.216028 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:16:45.216083 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:16:45.220385 (+ 4302us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:16:45.221057 (+ 672us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:16:45.221070 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:16:45.221074 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:16:45.221312 (+ 238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:16:45.221693 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:16:45.221699 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:16:45.221703 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:16:45.222262 (+ 559us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:16:45.222619 (+ 357us) server_negotiation.cc:1035] Waiting for connection context
0620 20:16:45.222784 (+ 165us) server_negotiation.cc:299] Negotiation successful
0620 20:16:45.223034 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":425}
I20260620 20:16:45.224157 25063 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:16:45.369513 17252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu with pid 24931
I20260620 20:16:45.395900 17252 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu pid 25072
Jun 20 20:16:45 dist-test-slave-2hpg krb5kdc[24814](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986604, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.16.217.2@KRBTEST.COM
I20260620 20:16:45.411345 25451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:45.396795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38746 (local address 127.16.217.2:36825)
0620 20:16:45.396973 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0620 20:16:45.396981 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 20:16:45.397004 (+ 23us) server_negotiation.cc:407] Connection header received
0620 20:16:45.397220 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:16:45.397226 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:16:45.397302 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:16:45.397406 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:16:45.398709 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:45.399410 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:45.400669 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:45.401092 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:45.401219 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:45.405522 (+ 4303us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:16:45.405534 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:16:45.405548 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:16:45.405579 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:16:45.408407 (+ 2828us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:16:45.409037 (+ 630us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:16:45.409045 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:16:45.409048 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:16:45.409169 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:16:45.409531 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:16:45.409537 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:16:45.409540 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:16:45.410248 (+ 708us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:16:45.410565 (+ 317us) server_negotiation.cc:1035] Waiting for connection context
0620 20:16:45.410819 (+ 254us) server_negotiation.cc:299] Negotiation successful
0620 20:16:45.411033 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 20:16:45.412106 25203 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:16:45.509087 17252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu with pid 25072
I20260620 20:16:45.527478 17252 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu pid 25212
Jun 20 20:16:45 dist-test-slave-2hpg krb5kdc[24814](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986604, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.16.217.3@KRBTEST.COM
I20260620 20:16:45.544335 25455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:45.528313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49798 (local address 127.16.217.3:44465)
0620 20:16:45.528515 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0620 20:16:45.528520 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:16:45.528543 (+ 23us) server_negotiation.cc:407] Connection header received
0620 20:16:45.528610 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:16:45.528615 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:16:45.528688 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:16:45.528806 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:16:45.530088 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:45.530865 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:45.532101 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:45.532570 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:45.532704 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:45.537518 (+ 4814us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:16:45.537531 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:16:45.537548 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:16:45.537593 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:16:45.541109 (+ 3516us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:16:45.542128 (+ 1019us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:16:45.542139 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:16:45.542144 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:16:45.542299 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:16:45.542770 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:16:45.542774 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:16:45.542776 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:16:45.543138 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:16:45.543365 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0620 20:16:45.543859 (+ 494us) server_negotiation.cc:299] Negotiation successful
0620 20:16:45.544024 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 20:16:45.545116 25343 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:16:45.677934 17252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu with pid 25212
I20260620 20:16:45.698832 17252 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu pid 24827
I20260620 20:16:45.700138 24900 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:16:45.814108 17252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu with pid 24827
2026-06-20T20:16:45Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (6666 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781986536687152-17252-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 20:16:45 dist-test-slave-2hpg krb5kdc[25473](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 20:16:45 dist-test-slave-2hpg krb5kdc[25473](info): set up 2 sockets
Jun 20 20:16:45 dist-test-slave-2hpg krb5kdc[25473](info): commencing operation
krb5kdc: starting...
W20260620 20:16:46.915380 17252 mini_kdc.cc:121] Time spent starting KDC: real 1.055s user 0.003s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781986536687152-17252-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781986536687152-17252-0/krb5kdc/test-user.keytab.
Jun 20 20:16:47 dist-test-slave-2hpg krb5kdc[25473](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986607, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T20:16:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T20:16:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.217.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.217.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.217.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.62.keytab.
Entry for principal kudu/127.16.217.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.62.keytab.
I20260620 20:16:47.135493 17252 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
/tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781986536687152-17252-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781986536687152-17252-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781986536687152-17252-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781986536687152-17252-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.217.62:46731
--webserver_interface=127.16.217.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.62.keytab
--principal=kudu/127.16.217.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.217.20:38715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.217.62:46731
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781986536687152-17252-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781986536687152-17252-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781986536687152-17252-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:16:47.461776 25486 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:16:47.462080 25486 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:16:47.462242 25486 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:16:47.469496 25486 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:16:47.469606 25486 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:16:47.469694 25486 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:16:47.469785 25486 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:16:47.469861 25486 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:16:47.479875 25486 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.217.20:38715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781986536687152-17252-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781986536687152-17252-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.217.62:46731
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.217.62:46731
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781986536687152-17252-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.217.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781986536687152-17252-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:16:47.481268 25486 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:16:47.482640 25486 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:16:47.494920 25491 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:16:47.494961 25494 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:16:47.495035 25492 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:16:47.495769 25486 server_base.cc:1034] running on GCE node
I20260620 20:16:47.496546 25486 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:16:47.498020 25486 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:16:47.499254 25486 hybrid_clock.cc:648] HybridClock initialized: now 1781986607499190 us; error 51 us; skew 500 ppm
I20260620 20:16:47.499591 25486 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-06-20T20:16:47Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1746 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986536687152-17252-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 20:16:47 dist-test-slave-2hpg krb5kdc[25500](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 20:16:47 dist-test-slave-2hpg krb5kdc[25500](info): set up 2 sockets
Jun 20 20:16:47 dist-test-slave-2hpg krb5kdc[25500](info): commencing operation
krb5kdc: starting...
W20260620 20:16:48.667131 17252 mini_kdc.cc:121] Time spent starting KDC: real 1.061s user 0.002s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986536687152-17252-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986536687152-17252-0/krb5kdc/test-user.keytab.
Jun 20 20:16:48 dist-test-slave-2hpg krb5kdc[25500](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986608, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T20:16:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T20:16:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.217.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.217.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.217.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.62.keytab.
Entry for principal kudu/127.16.217.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.62.keytab.
I20260620 20:16:48.883064 17252 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
/tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986536687152-17252-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986536687152-17252-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986536687152-17252-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986536687152-17252-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.217.62:44407
--webserver_interface=127.16.217.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.62.keytab
--principal=kudu/127.16.217.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.217.20:46329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.217.62:44407
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986536687152-17252-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986536687152-17252-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986536687152-17252-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:16:49.218391 25513 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:16:49.218709 25513 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:16:49.218864 25513 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:16:49.226310 25513 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:16:49.226428 25513 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:16:49.226560 25513 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:16:49.226651 25513 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:16:49.226749 25513 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:16:49.237367 25513 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.217.20:46329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986536687152-17252-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986536687152-17252-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.217.62:44407
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.62.keytab
--principal=kudu/127.16.217.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.217.62:44407
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986536687152-17252-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.217.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986536687152-17252-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:16:49.238778 25513 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:16:49.240096 25513 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:16:49.251916 25518 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:16:49.251895 25521 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:16:49.251852 25519 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:16:49.253127 25513 server_base.cc:1034] running on GCE node
I20260620 20:16:49.254040 25513 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:16:49.255411 25513 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:16:49.256644 25513 hybrid_clock.cc:648] HybridClock initialized: now 1781986609256601 us; error 56 us; skew 500 ppm
I20260620 20:16:49.256989 25513 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:16:49 dist-test-slave-2hpg krb5kdc[25500](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986609, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.217.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:16:49.264251 25513 init.cc:377] Logged in from keytab as kudu/127.16.217.62@KRBTEST.COM (short username kudu)
I20260620 20:16:49.265820 25513 webserver.cc:460] Webserver started at http://127.16.217.62:35291/ using document root <none> and password file <none>
I20260620 20:16:49.266562 25513 fs_manager.cc:359] Metadata directory not provided
I20260620 20:16:49.266708 25513 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:16:49.267021 25513 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:16:49.270009 25513 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986536687152-17252-0/minicluster-data/master-0/data/instance:
uuid: "559f4d0a0aaf425d97cb253a5c385f07"
format_stamp: "Formatted at 2026-06-20 20:16:49 on dist-test-slave-2hpg"
server_key: "a06c9b39d789e1424f18042494063183"
server_key_iv: "ec1645d01090ce77c07a6f98ccb2878b"
server_key_version: "encryptionkey@0"
I20260620 20:16:49.270766 25513 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986536687152-17252-0/minicluster-data/master-0/wal/instance:
uuid: "559f4d0a0aaf425d97cb253a5c385f07"
format_stamp: "Formatted at 2026-06-20 20:16:49 on dist-test-slave-2hpg"
server_key: "a06c9b39d789e1424f18042494063183"
server_key_iv: "ec1645d01090ce77c07a6f98ccb2878b"
server_key_version: "encryptionkey@0"
I20260620 20:16:49.275897 25513 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260620 20:16:49.279405 25528 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:16:49.280705 25513 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 20:16:49.280903 25513 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986536687152-17252-0/minicluster-data/master-0/data,/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986536687152-17252-0/minicluster-data/master-0/wal
uuid: "559f4d0a0aaf425d97cb253a5c385f07"
format_stamp: "Formatted at 2026-06-20 20:16:49 on dist-test-slave-2hpg"
server_key: "a06c9b39d789e1424f18042494063183"
server_key_iv: "ec1645d01090ce77c07a6f98ccb2878b"
server_key_version: "encryptionkey@0"
I20260620 20:16:49.281103 25513 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986536687152-17252-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986536687152-17252-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986536687152-17252-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:16:49.302930 25513 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:16:49.307246 25513 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:16:49.307531 25513 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:16:49.326136 25513 rpc_server.cc:307] RPC server started. Bound to: 127.16.217.62:44407
I20260620 20:16:49.326187 25579 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.217.62:44407 every 8 connection(s)
I20260620 20:16:49.327682 25513 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986536687152-17252-0/minicluster-data/master-0/data/info.pb
I20260620 20:16:49.331403 25580 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:16:49.334197 17252 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu as pid 25513
I20260620 20:16:49.334400 17252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986536687152-17252-0/minicluster-data/master-0/wal/instance
I20260620 20:16:49.335173 17252 external_mini_cluster.cc:1428] Setting key 8a46b113fda3cb6865322e0ebe2c1ba9
I20260620 20:16:49.343979 25580 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 559f4d0a0aaf425d97cb253a5c385f07: Bootstrap starting.
I20260620 20:16:49.347721 25580 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 559f4d0a0aaf425d97cb253a5c385f07: Neither blocks nor log segments found. Creating new log.
I20260620 20:16:49.349021 25580 log.cc:824] T 00000000000000000000000000000000 P 559f4d0a0aaf425d97cb253a5c385f07: Log is configured to *not* fsync() on all Append() calls
Jun 20 20:16:49 dist-test-slave-2hpg krb5kdc[25500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986608, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.217.62@KRBTEST.COM
I20260620 20:16:49.351989 25580 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 559f4d0a0aaf425d97cb253a5c385f07: No bootstrap required, opened a new log
I20260620 20:16:49.358158 25583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:49.337235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50776 (local address 127.16.217.62:44407)
0620 20:16:49.338037 (+ 802us) server_negotiation.cc:206] Beginning negotiation
0620 20:16:49.338049 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 20:16:49.338102 (+ 53us) server_negotiation.cc:407] Connection header received
0620 20:16:49.339751 (+ 1649us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:16:49.339772 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:16:49.340156 (+ 384us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:16:49.340485 (+ 329us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:16:49.342321 (+ 1836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:49.343589 (+ 1268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:49.345238 (+ 1649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:49.345961 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:49.346096 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:49.350821 (+ 4725us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:16:49.350829 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:16:49.350843 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:16:49.350879 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:16:49.354055 (+ 3176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:16:49.354881 (+ 826us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:16:49.354890 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:16:49.354897 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:16:49.355052 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:16:49.355460 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:16:49.355466 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:16:49.355468 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:16:49.355982 (+ 514us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:16:49.356186 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0620 20:16:49.356662 (+ 476us) server_negotiation.cc:299] Negotiation successful
0620 20:16:49.357048 (+ 386us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":569,"thread_start_us":363,"threads_started":1}
I20260620 20:16:49.361339 25580 raft_consensus.cc:348] T 00000000000000000000000000000000 P 559f4d0a0aaf425d97cb253a5c385f07 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "559f4d0a0aaf425d97cb253a5c385f07" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 44407 } }
I20260620 20:16:49.361896 25580 raft_consensus.cc:374] T 00000000000000000000000000000000 P 559f4d0a0aaf425d97cb253a5c385f07 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:16:49.362036 25580 raft_consensus.cc:729] T 00000000000000000000000000000000 P 559f4d0a0aaf425d97cb253a5c385f07 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 559f4d0a0aaf425d97cb253a5c385f07, State: Initialized, Role: FOLLOWER
I20260620 20:16:49.362614 25580 consensus_queue.cc:260] T 00000000000000000000000000000000 P 559f4d0a0aaf425d97cb253a5c385f07 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "559f4d0a0aaf425d97cb253a5c385f07" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 44407 } }
I20260620 20:16:49.362861 25580 raft_consensus.cc:388] T 00000000000000000000000000000000 P 559f4d0a0aaf425d97cb253a5c385f07 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 20:16:49.362967 25580 raft_consensus.cc:482] T 00000000000000000000000000000000 P 559f4d0a0aaf425d97cb253a5c385f07 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 20:16:49.363094 25580 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 559f4d0a0aaf425d97cb253a5c385f07 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:16:49.365278 25580 raft_consensus.cc:504] T 00000000000000000000000000000000 P 559f4d0a0aaf425d97cb253a5c385f07 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "559f4d0a0aaf425d97cb253a5c385f07" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 44407 } }
I20260620 20:16:49.365736 25580 leader_election.cc:302] T 00000000000000000000000000000000 P 559f4d0a0aaf425d97cb253a5c385f07 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 559f4d0a0aaf425d97cb253a5c385f07; no voters:
I20260620 20:16:49.366452 25580 leader_election.cc:288] T 00000000000000000000000000000000 P 559f4d0a0aaf425d97cb253a5c385f07 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 20:16:49.366596 25585 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 559f4d0a0aaf425d97cb253a5c385f07 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:16:49.367348 25585 raft_consensus.cc:686] T 00000000000000000000000000000000 P 559f4d0a0aaf425d97cb253a5c385f07 [term 1 LEADER]: Becoming Leader. State: Replica: 559f4d0a0aaf425d97cb253a5c385f07, State: Running, Role: LEADER
I20260620 20:16:49.367841 25585 consensus_queue.cc:237] T 00000000000000000000000000000000 P 559f4d0a0aaf425d97cb253a5c385f07 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "559f4d0a0aaf425d97cb253a5c385f07" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 44407 } }
I20260620 20:16:49.368631 25580 sys_catalog.cc:564] T 00000000000000000000000000000000 P 559f4d0a0aaf425d97cb253a5c385f07 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:16:49.370640 25586 sys_catalog.cc:455] T 00000000000000000000000000000000 P 559f4d0a0aaf425d97cb253a5c385f07 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "559f4d0a0aaf425d97cb253a5c385f07" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "559f4d0a0aaf425d97cb253a5c385f07" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 44407 } } }
I20260620 20:16:49.371191 25586 sys_catalog.cc:458] T 00000000000000000000000000000000 P 559f4d0a0aaf425d97cb253a5c385f07 [sys.catalog]: This master's current role is: LEADER
I20260620 20:16:49.371873 25587 sys_catalog.cc:455] T 00000000000000000000000000000000 P 559f4d0a0aaf425d97cb253a5c385f07 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 559f4d0a0aaf425d97cb253a5c385f07. Latest consensus state: current_term: 1 leader_uuid: "559f4d0a0aaf425d97cb253a5c385f07" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "559f4d0a0aaf425d97cb253a5c385f07" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 44407 } } }
I20260620 20:16:49.372351 25587 sys_catalog.cc:458] T 00000000000000000000000000000000 P 559f4d0a0aaf425d97cb253a5c385f07 [sys.catalog]: This master's current role is: LEADER
I20260620 20:16:49.372920 25593 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 20:16:49.378466 25593 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 20:16:49.387988 25593 catalog_manager.cc:1347] Generated new cluster ID: 7971ae4bc0514c74b5d9e415f0321a31
I20260620 20:16:49.388146 25593 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 20:16:49.420238 25593 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 20:16:49.421595 25593 catalog_manager.cc:1504] Loading token signing keys...
I20260620 20:16:49.428229 25593 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 559f4d0a0aaf425d97cb253a5c385f07: Generated new TSK 0
I20260620 20:16:49.429057 25593 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.217.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.217.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.217.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.1.keytab.
Entry for principal kudu/127.16.217.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.1.keytab.
I20260620 20:16:49.487126 17252 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
/tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986536687152-17252-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986536687152-17252-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986536687152-17252-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986536687152-17252-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.217.1:0
--local_ip_for_outbound_sockets=127.16.217.1
--webserver_interface=127.16.217.1
--webserver_port=0
--tserver_master_addrs=127.16.217.62:44407
--keytab_file=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.1.keytab
--principal=kudu/127.16.217.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.217.20:46329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986536687152-17252-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986536687152-17252-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986536687152-17252-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:16:49.816656 25606 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:16:49.816944 25606 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:16:49.817039 25606 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:16:49.824131 25606 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:16:49.824244 25606 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:16:49.824357 25606 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.217.1
I20260620 20:16:49.833927 25606 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.217.20:46329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986536687152-17252-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986536687152-17252-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.1.keytab
--principal=kudu/127.16.217.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.217.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986536687152-17252-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.217.1
--webserver_port=0
--tserver_master_addrs=127.16.217.62:44407
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.217.1
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986536687152-17252-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:16:49.835117 25606 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:16:49.836422 25606 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:16:49.849229 25611 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:16:49.849275 25612 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:16:49.849314 25614 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:16:49.850066 25606 server_base.cc:1034] running on GCE node
I20260620 20:16:49.850739 25606 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:16:49.851662 25606 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:16:49.852953 25606 hybrid_clock.cc:648] HybridClock initialized: now 1781986609852888 us; error 48 us; skew 500 ppm
I20260620 20:16:49.853296 25606 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:16:49 dist-test-slave-2hpg krb5kdc[25500](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986609, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.217.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:16:49.856598 25606 init.cc:377] Logged in from keytab as kudu/127.16.217.1@KRBTEST.COM (short username kudu)
I20260620 20:16:49.858428 25606 webserver.cc:460] Webserver started at http://127.16.217.1:36291/ using document root <none> and password file <none>
I20260620 20:16:49.859099 25606 fs_manager.cc:359] Metadata directory not provided
I20260620 20:16:49.859247 25606 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:16:49.859573 25606 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:16:49.862618 25606 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986536687152-17252-0/minicluster-data/ts-0/data/instance:
uuid: "ae410d45ecf2416a9c6bb6a7029fc628"
format_stamp: "Formatted at 2026-06-20 20:16:49 on dist-test-slave-2hpg"
server_key: "ad0f55877da8a0dede46c7ae2e8e970e"
server_key_iv: "3cd1a77ec4f0e25d8236115d5210c16b"
server_key_version: "encryptionkey@0"
I20260620 20:16:49.863344 25606 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986536687152-17252-0/minicluster-data/ts-0/wal/instance:
uuid: "ae410d45ecf2416a9c6bb6a7029fc628"
format_stamp: "Formatted at 2026-06-20 20:16:49 on dist-test-slave-2hpg"
server_key: "ad0f55877da8a0dede46c7ae2e8e970e"
server_key_iv: "3cd1a77ec4f0e25d8236115d5210c16b"
server_key_version: "encryptionkey@0"
I20260620 20:16:49.868505 25606 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 20:16:49.872100 25621 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:16:49.873533 25606 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 20:16:49.873709 25606 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986536687152-17252-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986536687152-17252-0/minicluster-data/ts-0/wal
uuid: "ae410d45ecf2416a9c6bb6a7029fc628"
format_stamp: "Formatted at 2026-06-20 20:16:49 on dist-test-slave-2hpg"
server_key: "ad0f55877da8a0dede46c7ae2e8e970e"
server_key_iv: "3cd1a77ec4f0e25d8236115d5210c16b"
server_key_version: "encryptionkey@0"
I20260620 20:16:49.873937 25606 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986536687152-17252-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986536687152-17252-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986536687152-17252-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:16:49.886915 25606 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:16:49.891053 25606 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:16:49.891288 25606 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:16:49.892194 25606 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:16:49.893477 25606 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:16:49.893548 25606 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:16:49.893630 25606 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:16:49.893675 25606 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:16:49.926191 25606 rpc_server.cc:307] RPC server started. Bound to: 127.16.217.1:34899
I20260620 20:16:49.926241 25733 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.217.1:34899 every 8 connection(s)
I20260620 20:16:49.927553 25606 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986536687152-17252-0/minicluster-data/ts-0/data/info.pb
I20260620 20:16:49.928231 17252 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu as pid 25606
I20260620 20:16:49.928431 17252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986536687152-17252-0/minicluster-data/ts-0/wal/instance
I20260620 20:16:49.929299 17252 external_mini_cluster.cc:1428] Setting key 87257fad57828af4f46ced8404a4bd24
Jun 20 20:16:49 dist-test-slave-2hpg krb5kdc[25500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986609, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.217.1@KRBTEST.COM for kudu/127.16.217.62@KRBTEST.COM
I20260620 20:16:49.947896 25737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:49.931638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.217.1:45129 (local address 127.16.217.62:44407)
0620 20:16:49.932094 (+ 456us) server_negotiation.cc:206] Beginning negotiation
0620 20:16:49.932103 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 20:16:49.932230 (+ 127us) server_negotiation.cc:407] Connection header received
0620 20:16:49.934078 (+ 1848us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:16:49.934083 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:16:49.934173 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:16:49.934285 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:16:49.935739 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:49.936452 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:49.938046 (+ 1594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:49.938468 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:49.938603 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:49.942321 (+ 3718us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:16:49.942326 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:16:49.942330 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:16:49.942359 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:16:49.944802 (+ 2443us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:16:49.945548 (+ 746us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:16:49.945560 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:16:49.945563 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:16:49.945674 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:16:49.946173 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:16:49.946182 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:16:49.946184 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:16:49.946451 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:16:49.946603 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0620 20:16:49.947290 (+ 687us) server_negotiation.cc:299] Negotiation successful
0620 20:16:49.947535 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":335,"thread_start_us":919,"threads_started":1}
I20260620 20:16:49.948953 25736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:49.931214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:44407 (local address 127.16.217.1:45129)
0620 20:16:49.932065 (+ 851us) negotiation.cc:107] Waiting for socket to connect
0620 20:16:49.932114 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0620 20:16:49.933769 (+ 1655us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:16:49.934520 (+ 751us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:16:49.934529 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:16:49.935013 (+ 484us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:16:49.935568 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:16:49.935583 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:16:49.936605 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:16:49.936611 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:16:49.937835 (+ 1224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:16:49.937846 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:16:49.938699 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:16:49.938707 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:16:49.938949 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:49.939640 (+ 691us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:16:49.939661 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:16:49.942087 (+ 2426us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:16:49.944944 (+ 2857us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:16:49.944953 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:16:49.944971 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:16:49.945422 (+ 451us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:16:49.945822 (+ 400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:16:49.945829 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:16:49.945833 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:16:49.946055 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:16:49.946572 (+ 517us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:16:49.946580 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:16:49.946922 (+ 342us) client_negotiation.cc:769] Sending connection context
0620 20:16:49.947176 (+ 254us) client_negotiation.cc:240] Negotiation successful
0620 20:16:49.947738 (+ 562us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":501,"thread_start_us":495,"threads_started":1}
WARNING: no policy specified for kudu/127.16.217.2@KRBTEST.COM; defaulting to no policy
I20260620 20:16:49.950634 25734 heartbeater.cc:347] Connected to a master server at 127.16.217.62:44407
I20260620 20:16:49.950948 25734 heartbeater.cc:464] Registering TS with master...
I20260620 20:16:49.951735 25734 heartbeater.cc:511] Master 127.16.217.62:44407 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.217.2@KRBTEST.COM" created.
I20260620 20:16:49.954150 25545 ts_manager.cc:194] Registered new tserver with Master: ae410d45ecf2416a9c6bb6a7029fc628 (127.16.217.1:34899)
I20260620 20:16:49.956296 25545 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.217.1@KRBTEST.COM'} at 127.16.217.1:45129
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.217.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.2.keytab.
Entry for principal kudu/127.16.217.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.2.keytab.
I20260620 20:16:49.980989 17252 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
/tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986536687152-17252-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986536687152-17252-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986536687152-17252-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986536687152-17252-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.217.2:0
--local_ip_for_outbound_sockets=127.16.217.2
--webserver_interface=127.16.217.2
--webserver_port=0
--tserver_master_addrs=127.16.217.62:44407
--keytab_file=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.2.keytab
--principal=kudu/127.16.217.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.217.20:46329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986536687152-17252-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986536687152-17252-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986536687152-17252-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:16:50.312201 25740 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:16:50.312502 25740 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:16:50.312608 25740 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:16:50.320081 25740 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:16:50.320217 25740 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:16:50.320339 25740 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.217.2
I20260620 20:16:50.334082 25740 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.217.20:46329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986536687152-17252-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986536687152-17252-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.2.keytab
--principal=kudu/127.16.217.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.217.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986536687152-17252-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.217.2
--webserver_port=0
--tserver_master_addrs=127.16.217.62:44407
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.217.2
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986536687152-17252-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:16:50.335664 25740 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:16:50.337409 25740 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:16:50.352453 25745 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:16:50.352538 25746 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:16:50.352497 25748 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:16:50.353696 25740 server_base.cc:1034] running on GCE node
I20260620 20:16:50.354463 25740 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:16:50.355381 25740 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:16:50.356600 25740 hybrid_clock.cc:648] HybridClock initialized: now 1781986610356512 us; error 81 us; skew 500 ppm
I20260620 20:16:50.356930 25740 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:16:50 dist-test-slave-2hpg krb5kdc[25500](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986610, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.217.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:16:50.359987 25740 init.cc:377] Logged in from keytab as kudu/127.16.217.2@KRBTEST.COM (short username kudu)
I20260620 20:16:50.361366 25740 webserver.cc:460] Webserver started at http://127.16.217.2:37481/ using document root <none> and password file <none>
I20260620 20:16:50.362121 25740 fs_manager.cc:359] Metadata directory not provided
I20260620 20:16:50.362258 25740 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:16:50.362552 25740 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:16:50.365329 25740 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986536687152-17252-0/minicluster-data/ts-1/data/instance:
uuid: "b3e9ff6abe074399a453e354224a28fa"
format_stamp: "Formatted at 2026-06-20 20:16:50 on dist-test-slave-2hpg"
server_key: "cc445566fc6fbcf5cd0ac1e654e345f5"
server_key_iv: "bacb0089de00def6f993f000a86fda52"
server_key_version: "encryptionkey@0"
I20260620 20:16:50.366168 25740 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986536687152-17252-0/minicluster-data/ts-1/wal/instance:
uuid: "b3e9ff6abe074399a453e354224a28fa"
format_stamp: "Formatted at 2026-06-20 20:16:50 on dist-test-slave-2hpg"
server_key: "cc445566fc6fbcf5cd0ac1e654e345f5"
server_key_iv: "bacb0089de00def6f993f000a86fda52"
server_key_version: "encryptionkey@0"
I20260620 20:16:50.371187 25740 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260620 20:16:50.374694 25755 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:16:50.376030 25740 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 20:16:50.376189 25740 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986536687152-17252-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986536687152-17252-0/minicluster-data/ts-1/wal
uuid: "b3e9ff6abe074399a453e354224a28fa"
format_stamp: "Formatted at 2026-06-20 20:16:50 on dist-test-slave-2hpg"
server_key: "cc445566fc6fbcf5cd0ac1e654e345f5"
server_key_iv: "bacb0089de00def6f993f000a86fda52"
server_key_version: "encryptionkey@0"
I20260620 20:16:50.376411 25740 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986536687152-17252-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986536687152-17252-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986536687152-17252-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:16:50.399688 25740 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:16:50.403750 25740 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:16:50.404050 25740 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:16:50.404994 25740 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:16:50.406384 25740 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:16:50.406482 25740 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:16:50.406625 25740 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:16:50.406692 25740 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:16:50.439393 25740 rpc_server.cc:307] RPC server started. Bound to: 127.16.217.2:33487
I20260620 20:16:50.439481 25867 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.217.2:33487 every 8 connection(s)
I20260620 20:16:50.440702 25740 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986536687152-17252-0/minicluster-data/ts-1/data/info.pb
I20260620 20:16:50.443277 17252 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu as pid 25740
I20260620 20:16:50.443457 17252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986536687152-17252-0/minicluster-data/ts-1/wal/instance
I20260620 20:16:50.443907 17252 external_mini_cluster.cc:1428] Setting key e66e7f4cd64596dfe720ebcc7ec96fdf
Jun 20 20:16:50 dist-test-slave-2hpg krb5kdc[25500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986610, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.217.2@KRBTEST.COM for kudu/127.16.217.62@KRBTEST.COM
I20260620 20:16:50.465344 25737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:50.443392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.217.2:51805 (local address 127.16.217.62:44407)
0620 20:16:50.443609 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0620 20:16:50.443616 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:16:50.445396 (+ 1780us) server_negotiation.cc:407] Connection header received
0620 20:16:50.448023 (+ 2627us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:16:50.448031 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:16:50.448106 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:16:50.448188 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:16:50.450379 (+ 2191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:50.451049 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:50.453623 (+ 2574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:50.454254 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:50.454397 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:50.457911 (+ 3514us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:16:50.457921 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:16:50.457930 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:16:50.457974 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:16:50.461581 (+ 3607us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:16:50.462461 (+ 880us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:16:50.462468 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:16:50.462475 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:16:50.462623 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:16:50.463249 (+ 626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:16:50.463257 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:16:50.463263 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:16:50.463652 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:16:50.463820 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0620 20:16:50.464620 (+ 800us) server_negotiation.cc:299] Negotiation successful
0620 20:16:50.464877 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260620 20:16:50.466064 25870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:50.443936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:44407 (local address 127.16.217.2:51805)
0620 20:16:50.445205 (+ 1269us) negotiation.cc:107] Waiting for socket to connect
0620 20:16:50.445270 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0620 20:16:50.447687 (+ 2417us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:16:50.448476 (+ 789us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:16:50.448492 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:16:50.449293 (+ 801us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:16:50.450198 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:16:50.450227 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:16:50.451229 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:16:50.451238 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:16:50.453461 (+ 2223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:16:50.453479 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:16:50.454485 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:16:50.454490 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:16:50.454691 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:50.455421 (+ 730us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:16:50.455442 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:16:50.457658 (+ 2216us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:16:50.461739 (+ 4081us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:16:50.461746 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:16:50.461759 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:16:50.462336 (+ 577us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:16:50.462907 (+ 571us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:16:50.462911 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:16:50.462913 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:16:50.463106 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:16:50.463780 (+ 674us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:16:50.463786 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:16:50.464227 (+ 441us) client_negotiation.cc:769] Sending connection context
0620 20:16:50.464465 (+ 238us) client_negotiation.cc:240] Negotiation successful
0620 20:16:50.464851 (+ 386us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":923,"thread_start_us":361,"threads_started":1}
I20260620 20:16:50.467658 25868 heartbeater.cc:347] Connected to a master server at 127.16.217.62:44407
I20260620 20:16:50.467996 25868 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.16.217.3@KRBTEST.COM; defaulting to no policy
I20260620 20:16:50.468858 25868 heartbeater.cc:511] Master 127.16.217.62:44407 requested a full tablet report, sending...
I20260620 20:16:50.470557 25545 ts_manager.cc:194] Registered new tserver with Master: b3e9ff6abe074399a453e354224a28fa (127.16.217.2:33487)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.217.3@KRBTEST.COM" created.
I20260620 20:16:50.471680 25545 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.217.2@KRBTEST.COM'} at 127.16.217.2:51805
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.217.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.3.keytab.
Entry for principal kudu/127.16.217.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.3.keytab.
I20260620 20:16:50.496685 17252 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
/tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986536687152-17252-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986536687152-17252-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986536687152-17252-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986536687152-17252-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.217.3:0
--local_ip_for_outbound_sockets=127.16.217.3
--webserver_interface=127.16.217.3
--webserver_port=0
--tserver_master_addrs=127.16.217.62:44407
--keytab_file=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.3.keytab
--principal=kudu/127.16.217.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.217.20:46329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986536687152-17252-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986536687152-17252-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986536687152-17252-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:16:50.810941 25873 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:16:50.811213 25873 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:16:50.811368 25873 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:16:50.818616 25873 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:16:50.818719 25873 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:16:50.818889 25873 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.217.3
I20260620 20:16:50.828563 25873 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.217.20:46329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986536687152-17252-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986536687152-17252-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.3.keytab
--principal=kudu/127.16.217.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.217.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986536687152-17252-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.217.3
--webserver_port=0
--tserver_master_addrs=127.16.217.62:44407
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.217.3
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986536687152-17252-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:16:50.829833 25873 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:16:50.831193 25873 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:16:50.844013 25878 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:16:50.844077 25879 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:16:50.844070 25881 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:16:50.844877 25873 server_base.cc:1034] running on GCE node
I20260620 20:16:50.845489 25873 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:16:50.846418 25873 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:16:50.847642 25873 hybrid_clock.cc:648] HybridClock initialized: now 1781986610847608 us; error 43 us; skew 500 ppm
I20260620 20:16:50.847963 25873 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:16:50 dist-test-slave-2hpg krb5kdc[25500](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986610, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.217.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:16:50.851141 25873 init.cc:377] Logged in from keytab as kudu/127.16.217.3@KRBTEST.COM (short username kudu)
I20260620 20:16:50.852573 25873 webserver.cc:460] Webserver started at http://127.16.217.3:35641/ using document root <none> and password file <none>
I20260620 20:16:50.853245 25873 fs_manager.cc:359] Metadata directory not provided
I20260620 20:16:50.853412 25873 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:16:50.853730 25873 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:16:50.856635 25873 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986536687152-17252-0/minicluster-data/ts-2/data/instance:
uuid: "eba861f191d54cbb88cb4c9f99f7a605"
format_stamp: "Formatted at 2026-06-20 20:16:50 on dist-test-slave-2hpg"
server_key: "2e371681a81728e83f024df35d9629b7"
server_key_iv: "c973f41ba2e8ee473b5ebd0f56b92587"
server_key_version: "encryptionkey@0"
I20260620 20:16:50.857487 25873 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986536687152-17252-0/minicluster-data/ts-2/wal/instance:
uuid: "eba861f191d54cbb88cb4c9f99f7a605"
format_stamp: "Formatted at 2026-06-20 20:16:50 on dist-test-slave-2hpg"
server_key: "2e371681a81728e83f024df35d9629b7"
server_key_iv: "c973f41ba2e8ee473b5ebd0f56b92587"
server_key_version: "encryptionkey@0"
I20260620 20:16:50.862896 25873 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260620 20:16:50.866389 25888 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:16:50.867689 25873 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 20:16:50.867877 25873 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986536687152-17252-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986536687152-17252-0/minicluster-data/ts-2/wal
uuid: "eba861f191d54cbb88cb4c9f99f7a605"
format_stamp: "Formatted at 2026-06-20 20:16:50 on dist-test-slave-2hpg"
server_key: "2e371681a81728e83f024df35d9629b7"
server_key_iv: "c973f41ba2e8ee473b5ebd0f56b92587"
server_key_version: "encryptionkey@0"
I20260620 20:16:50.868070 25873 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986536687152-17252-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986536687152-17252-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986536687152-17252-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:16:50.888475 25873 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:16:50.892745 25873 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:16:50.893054 25873 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:16:50.894093 25873 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:16:50.895444 25873 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:16:50.895557 25873 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:16:50.895730 25873 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:16:50.895805 25873 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:16:50.929342 25873 rpc_server.cc:307] RPC server started. Bound to: 127.16.217.3:36363
I20260620 20:16:50.929378 26000 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.217.3:36363 every 8 connection(s)
I20260620 20:16:50.930644 25873 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986536687152-17252-0/minicluster-data/ts-2/data/info.pb
I20260620 20:16:50.936659 17252 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu as pid 25873
I20260620 20:16:50.936838 17252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986536687152-17252-0/minicluster-data/ts-2/wal/instance
I20260620 20:16:50.937299 17252 external_mini_cluster.cc:1428] Setting key 041d3cab823d02c2152867d977bc039d
Jun 20 20:16:50 dist-test-slave-2hpg krb5kdc[25500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986610, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.217.3@KRBTEST.COM for kudu/127.16.217.62@KRBTEST.COM
I20260620 20:16:50.949779 25737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:50.933123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.217.3:50239 (local address 127.16.217.62:44407)
0620 20:16:50.933300 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0620 20:16:50.933304 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:16:50.934499 (+ 1195us) server_negotiation.cc:407] Connection header received
0620 20:16:50.936281 (+ 1782us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:16:50.936288 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:16:50.936373 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:16:50.936482 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:16:50.938121 (+ 1639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:50.939173 (+ 1052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:50.940757 (+ 1584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:50.941142 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:50.941260 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:50.944554 (+ 3294us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:16:50.944559 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:16:50.944563 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:16:50.944594 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:16:50.947009 (+ 2415us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:16:50.947660 (+ 651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:16:50.947664 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:16:50.947667 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:16:50.947797 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:16:50.948256 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:16:50.948259 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:16:50.948261 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:16:50.948627 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:16:50.948747 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0620 20:16:50.949358 (+ 611us) server_negotiation.cc:299] Negotiation successful
0620 20:16:50.949545 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 20:16:50.950670 26003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:50.933621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:44407 (local address 127.16.217.3:50239)
0620 20:16:50.934349 (+ 728us) negotiation.cc:107] Waiting for socket to connect
0620 20:16:50.934390 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 20:16:50.935990 (+ 1600us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:16:50.936699 (+ 709us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:16:50.936707 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:16:50.937289 (+ 582us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:16:50.937906 (+ 617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:16:50.937934 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:16:50.939340 (+ 1406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:16:50.939343 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:16:50.940629 (+ 1286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:16:50.940640 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:16:50.941341 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:16:50.941345 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:16:50.941539 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:50.942265 (+ 726us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:16:50.942299 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:16:50.944370 (+ 2071us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:16:50.947124 (+ 2754us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:16:50.947130 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:16:50.947147 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:16:50.947529 (+ 382us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:16:50.947898 (+ 369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:16:50.947902 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:16:50.947905 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:16:50.948101 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:16:50.948747 (+ 646us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:16:50.948753 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:16:50.949032 (+ 279us) client_negotiation.cc:769] Sending connection context
0620 20:16:50.949250 (+ 218us) client_negotiation.cc:240] Negotiation successful
0620 20:16:50.949596 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":508,"thread_start_us":310,"threads_started":1}
I20260620 20:16:50.952077 26001 heartbeater.cc:347] Connected to a master server at 127.16.217.62:44407
I20260620 20:16:50.952366 26001 heartbeater.cc:464] Registering TS with master...
I20260620 20:16:50.953070 26001 heartbeater.cc:511] Master 127.16.217.62:44407 requested a full tablet report, sending...
I20260620 20:16:50.954368 25545 ts_manager.cc:194] Registered new tserver with Master: eba861f191d54cbb88cb4c9f99f7a605 (127.16.217.3:36363)
I20260620 20:16:50.955178 25545 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.217.3@KRBTEST.COM'} at 127.16.217.3:50239
I20260620 20:16:50.959309 25734 heartbeater.cc:503] Master 127.16.217.62:44407 was elected leader, sending a full tablet report...
I20260620 20:16:50.964596 17252 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.16.217.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.16.217.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.16.217.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986536687152-17252-0/krb5kdc/oryx_127.16.217.1.keytab.
Entry for principal oryx/127.16.217.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986536687152-17252-0/krb5kdc/oryx_127.16.217.1.keytab.
WARNING: no policy specified for oryx/127.16.217.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.16.217.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.16.217.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986536687152-17252-0/krb5kdc/oryx_127.16.217.2.keytab.
Entry for principal oryx/127.16.217.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986536687152-17252-0/krb5kdc/oryx_127.16.217.2.keytab.
WARNING: no policy specified for oryx/127.16.217.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.16.217.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.16.217.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986536687152-17252-0/krb5kdc/oryx_127.16.217.3.keytab.
Entry for principal oryx/127.16.217.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986536687152-17252-0/krb5kdc/oryx_127.16.217.3.keytab.
I20260620 20:16:51.098894 17252 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu pid 25606
Jun 20 20:16:51 dist-test-slave-2hpg krb5kdc[25500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986608, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.217.1@KRBTEST.COM
I20260620 20:16:51.115135 26011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:51.099949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37284 (local address 127.16.217.1:34899)
0620 20:16:51.100627 (+ 678us) server_negotiation.cc:206] Beginning negotiation
0620 20:16:51.100636 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 20:16:51.100677 (+ 41us) server_negotiation.cc:407] Connection header received
0620 20:16:51.100749 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:16:51.100757 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:16:51.100990 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:16:51.101137 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:16:51.102524 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:51.103620 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:51.104872 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:51.105427 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:51.105527 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:51.109500 (+ 3973us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:16:51.109511 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:16:51.109524 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:16:51.109564 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:16:51.112297 (+ 2733us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:16:51.112968 (+ 671us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:16:51.112975 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:16:51.112977 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:16:51.113108 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:16:51.113450 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:16:51.113454 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:16:51.113456 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:16:51.113954 (+ 498us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:16:51.114200 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0620 20:16:51.114523 (+ 323us) server_negotiation.cc:299] Negotiation successful
0620 20:16:51.114772 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":521,"thread_start_us":324,"threads_started":1}
I20260620 20:16:51.116375 25729 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:16:51.200806 17252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu with pid 25606
I20260620 20:16:51.217262 17252 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu pid 25740
Jun 20 20:16:51 dist-test-slave-2hpg krb5kdc[25500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986608, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.217.2@KRBTEST.COM
I20260620 20:16:51.233253 26013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:51.218201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36756 (local address 127.16.217.2:33487)
0620 20:16:51.218720 (+ 519us) server_negotiation.cc:206] Beginning negotiation
0620 20:16:51.218729 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 20:16:51.218758 (+ 29us) server_negotiation.cc:407] Connection header received
0620 20:16:51.218816 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:16:51.218822 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:16:51.219032 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:16:51.219201 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:16:51.220428 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:51.221441 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:51.222800 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:51.223346 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:51.223484 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:51.227595 (+ 4111us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:16:51.227606 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:16:51.227622 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:16:51.227661 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:16:51.230495 (+ 2834us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:16:51.231174 (+ 679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:16:51.231181 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:16:51.231183 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:16:51.231309 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:16:51.231686 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:16:51.231696 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:16:51.231698 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:16:51.232027 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:16:51.232336 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0620 20:16:51.232639 (+ 303us) server_negotiation.cc:299] Negotiation successful
0620 20:16:51.232902 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":386,"thread_start_us":272,"threads_started":1}
I20260620 20:16:51.234602 25863 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:16:51.317335 17252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu with pid 25740
I20260620 20:16:51.333334 17252 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu pid 25873
Jun 20 20:16:51 dist-test-slave-2hpg krb5kdc[25500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986608, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.217.3@KRBTEST.COM
I20260620 20:16:51.349267 26015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:51.334423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47404 (local address 127.16.217.3:36363)
0620 20:16:51.335062 (+ 639us) server_negotiation.cc:206] Beginning negotiation
0620 20:16:51.335069 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:16:51.335099 (+ 30us) server_negotiation.cc:407] Connection header received
0620 20:16:51.335157 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:16:51.335163 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:16:51.335369 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:16:51.335500 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:16:51.336773 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:51.337822 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:51.339075 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:51.339608 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:51.339701 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:51.344003 (+ 4302us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:16:51.344013 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:16:51.344027 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:16:51.344063 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:16:51.346901 (+ 2838us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:16:51.347505 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:16:51.347512 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:16:51.347514 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:16:51.347634 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:16:51.347946 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:16:51.347950 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:16:51.347952 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:16:51.348263 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:16:51.348491 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0620 20:16:51.348691 (+ 200us) server_negotiation.cc:299] Negotiation successful
0620 20:16:51.348896 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":495,"thread_start_us":352,"threads_started":1}
I20260620 20:16:51.350816 25996 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:16:51.432979 17252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu with pid 25873
I20260620 20:16:51.448304 17252 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu pid 25513
I20260620 20:16:51.449481 25575 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:16:51.549304 17252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu with pid 25513
I20260620 20:16:51.569664 17252 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
/tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986536687152-17252-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986536687152-17252-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986536687152-17252-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986536687152-17252-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.217.62:44407
--webserver_interface=127.16.217.62
--webserver_port=35291
--keytab_file=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.62.keytab
--principal=kudu/127.16.217.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.217.20:46329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.217.62:44407
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986536687152-17252-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986536687152-17252-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986536687152-17252-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:16:51.885226 26018 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:16:51.885511 26018 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:16:51.885675 26018 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:16:51.892815 26018 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:16:51.892916 26018 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:16:51.893002 26018 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:16:51.893108 26018 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:16:51.893172 26018 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:16:51.902946 26018 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.217.20:46329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986536687152-17252-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986536687152-17252-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.217.62:44407
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.62.keytab
--principal=kudu/127.16.217.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.217.62:44407
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986536687152-17252-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.217.62
--webserver_port=35291
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986536687152-17252-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:16:51.904249 26018 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:16:51.905529 26018 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:16:51.917719 26026 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:16:51.917833 26023 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:16:51.917788 26024 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:16:51.919082 26018 server_base.cc:1034] running on GCE node
I20260620 20:16:51.919893 26018 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:16:51.921484 26018 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:16:51.922734 26018 hybrid_clock.cc:648] HybridClock initialized: now 1781986611922676 us; error 39 us; skew 500 ppm
I20260620 20:16:51.923112 26018 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:16:51 dist-test-slave-2hpg krb5kdc[25500](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986611, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.217.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:16:51.926250 26018 init.cc:377] Logged in from keytab as kudu/127.16.217.62@KRBTEST.COM (short username kudu)
I20260620 20:16:51.927754 26018 webserver.cc:460] Webserver started at http://127.16.217.62:35291/ using document root <none> and password file <none>
I20260620 20:16:51.928623 26018 fs_manager.cc:359] Metadata directory not provided
I20260620 20:16:51.928925 26018 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:16:51.934638 26018 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260620 20:16:51.937698 26033 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:16:51.939141 26018 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 20:16:51.939345 26018 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986536687152-17252-0/minicluster-data/master-0/data,/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986536687152-17252-0/minicluster-data/master-0/wal
uuid: "559f4d0a0aaf425d97cb253a5c385f07"
format_stamp: "Formatted at 2026-06-20 20:16:49 on dist-test-slave-2hpg"
server_key: "a06c9b39d789e1424f18042494063183"
server_key_iv: "ec1645d01090ce77c07a6f98ccb2878b"
server_key_version: "encryptionkey@0"
I20260620 20:16:51.940339 26018 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986536687152-17252-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986536687152-17252-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986536687152-17252-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:16:51.959789 26018 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:16:51.964457 26018 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:16:51.964775 26018 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:16:51.984352 26018 rpc_server.cc:307] RPC server started. Bound to: 127.16.217.62:44407
I20260620 20:16:51.984371 26084 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.217.62:44407 every 8 connection(s)
I20260620 20:16:51.986013 26018 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986536687152-17252-0/minicluster-data/master-0/data/info.pb
I20260620 20:16:51.989423 17252 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu as pid 26018
I20260620 20:16:51.990332 17252 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
/tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986536687152-17252-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986536687152-17252-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986536687152-17252-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986536687152-17252-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.217.1:34899
--local_ip_for_outbound_sockets=127.16.217.1
--tserver_master_addrs=127.16.217.62:44407
--webserver_port=36291
--webserver_interface=127.16.217.1
--keytab_file=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.1.keytab
--principal=kudu/127.16.217.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.217.20:46329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.16.217.1
--keytab_file=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986536687152-17252-0/krb5kdc/oryx_127.16.217.1.keytab with env {KRB5CCNAME=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986536687152-17252-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986536687152-17252-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986536687152-17252-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 20:16:51.992741 26085 sys_catalog.cc:263] Verifying existing consensus state
I20260620 20:16:51.995932 26085 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 559f4d0a0aaf425d97cb253a5c385f07: Bootstrap starting.
I20260620 20:16:52.023175 26085 log.cc:824] T 00000000000000000000000000000000 P 559f4d0a0aaf425d97cb253a5c385f07: Log is configured to *not* fsync() on all Append() calls
I20260620 20:16:52.033663 26085 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 559f4d0a0aaf425d97cb253a5c385f07: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 20:16:52.034286 26085 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 559f4d0a0aaf425d97cb253a5c385f07: Bootstrap complete.
I20260620 20:16:52.047581 26085 raft_consensus.cc:348] T 00000000000000000000000000000000 P 559f4d0a0aaf425d97cb253a5c385f07 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "559f4d0a0aaf425d97cb253a5c385f07" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 44407 } }
I20260620 20:16:52.048830 26085 raft_consensus.cc:729] T 00000000000000000000000000000000 P 559f4d0a0aaf425d97cb253a5c385f07 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 559f4d0a0aaf425d97cb253a5c385f07, State: Initialized, Role: FOLLOWER
I20260620 20:16:52.049592 26085 consensus_queue.cc:260] T 00000000000000000000000000000000 P 559f4d0a0aaf425d97cb253a5c385f07 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "559f4d0a0aaf425d97cb253a5c385f07" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 44407 } }
I20260620 20:16:52.049907 26085 raft_consensus.cc:388] T 00000000000000000000000000000000 P 559f4d0a0aaf425d97cb253a5c385f07 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 20:16:52.050031 26085 raft_consensus.cc:482] T 00000000000000000000000000000000 P 559f4d0a0aaf425d97cb253a5c385f07 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 20:16:52.050215 26085 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 559f4d0a0aaf425d97cb253a5c385f07 [term 1 FOLLOWER]: Advancing to term 2
I20260620 20:16:52.055968 26085 raft_consensus.cc:504] T 00000000000000000000000000000000 P 559f4d0a0aaf425d97cb253a5c385f07 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "559f4d0a0aaf425d97cb253a5c385f07" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 44407 } }
I20260620 20:16:52.056648 26085 leader_election.cc:302] T 00000000000000000000000000000000 P 559f4d0a0aaf425d97cb253a5c385f07 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 559f4d0a0aaf425d97cb253a5c385f07; no voters:
I20260620 20:16:52.057351 26085 leader_election.cc:288] T 00000000000000000000000000000000 P 559f4d0a0aaf425d97cb253a5c385f07 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 20:16:52.057492 26089 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 559f4d0a0aaf425d97cb253a5c385f07 [term 2 FOLLOWER]: Leader election won for term 2
I20260620 20:16:52.059100 26089 raft_consensus.cc:686] T 00000000000000000000000000000000 P 559f4d0a0aaf425d97cb253a5c385f07 [term 2 LEADER]: Becoming Leader. State: Replica: 559f4d0a0aaf425d97cb253a5c385f07, State: Running, Role: LEADER
I20260620 20:16:52.059643 26089 consensus_queue.cc:237] T 00000000000000000000000000000000 P 559f4d0a0aaf425d97cb253a5c385f07 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "559f4d0a0aaf425d97cb253a5c385f07" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 44407 } }
I20260620 20:16:52.059893 26085 sys_catalog.cc:564] T 00000000000000000000000000000000 P 559f4d0a0aaf425d97cb253a5c385f07 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:16:52.062080 26091 sys_catalog.cc:455] T 00000000000000000000000000000000 P 559f4d0a0aaf425d97cb253a5c385f07 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 559f4d0a0aaf425d97cb253a5c385f07. Latest consensus state: current_term: 2 leader_uuid: "559f4d0a0aaf425d97cb253a5c385f07" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "559f4d0a0aaf425d97cb253a5c385f07" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 44407 } } }
I20260620 20:16:52.062417 26091 sys_catalog.cc:458] T 00000000000000000000000000000000 P 559f4d0a0aaf425d97cb253a5c385f07 [sys.catalog]: This master's current role is: LEADER
I20260620 20:16:52.062911 26090 sys_catalog.cc:455] T 00000000000000000000000000000000 P 559f4d0a0aaf425d97cb253a5c385f07 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "559f4d0a0aaf425d97cb253a5c385f07" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "559f4d0a0aaf425d97cb253a5c385f07" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 44407 } } }
I20260620 20:16:52.063231 26090 sys_catalog.cc:458] T 00000000000000000000000000000000 P 559f4d0a0aaf425d97cb253a5c385f07 [sys.catalog]: This master's current role is: LEADER
I20260620 20:16:52.073566 26095 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 20:16:52.079146 26095 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 20:16:52.080482 26095 catalog_manager.cc:1259] Loaded cluster ID: 7971ae4bc0514c74b5d9e415f0321a31
I20260620 20:16:52.080575 26095 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 20:16:52.084661 26095 catalog_manager.cc:1504] Loading token signing keys...
I20260620 20:16:52.086054 26095 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 559f4d0a0aaf425d97cb253a5c385f07: Loaded TSK: 0
I20260620 20:16:52.087507 26095 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 20:16:52.342612 26087 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:16:52.342886 26087 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:16:52.343034 26087 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:16:52.350371 26087 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:16:52.350474 26087 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:16:52.350629 26087 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.217.1
I20260620 20:16:52.360311 26087 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.217.20:46329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986536687152-17252-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986536687152-17252-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986536687152-17252-0/krb5kdc/oryx_127.16.217.1.keytab
--principal=oryx/127.16.217.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.217.1:34899
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986536687152-17252-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.217.1
--webserver_port=36291
--tserver_master_addrs=127.16.217.62:44407
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.217.1
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986536687152-17252-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:16:52.361616 26087 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:16:52.362929 26087 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:16:52.376142 26112 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:16:52.376405 26114 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:16:52.376892 26111 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:16:52.377990 26087 server_base.cc:1034] running on GCE node
I20260620 20:16:52.378572 26087 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:16:52.379488 26087 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:16:52.380709 26087 hybrid_clock.cc:648] HybridClock initialized: now 1781986612380653 us; error 51 us; skew 500 ppm
I20260620 20:16:52.381023 26087 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:16:52 dist-test-slave-2hpg krb5kdc[25500](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986612, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.217.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:16:52.384124 26087 init.cc:377] Logged in from keytab as oryx/127.16.217.1@KRBTEST.COM (short username oryx)
I20260620 20:16:52.385601 26087 webserver.cc:460] Webserver started at http://127.16.217.1:36291/ using document root <none> and password file <none>
I20260620 20:16:52.386353 26087 fs_manager.cc:359] Metadata directory not provided
I20260620 20:16:52.386497 26087 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:16:52.391816 26087 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 20:16:52.394603 26121 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:16:52.396040 26087 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 20:16:52.396214 26087 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986536687152-17252-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986536687152-17252-0/minicluster-data/ts-0/wal
uuid: "ae410d45ecf2416a9c6bb6a7029fc628"
format_stamp: "Formatted at 2026-06-20 20:16:49 on dist-test-slave-2hpg"
server_key: "ad0f55877da8a0dede46c7ae2e8e970e"
server_key_iv: "3cd1a77ec4f0e25d8236115d5210c16b"
server_key_version: "encryptionkey@0"
I20260620 20:16:52.397176 26087 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986536687152-17252-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986536687152-17252-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986536687152-17252-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:16:52.436394 26087 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:16:52.440441 26087 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:16:52.440711 26087 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:16:52.441694 26087 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:16:52.443027 26087 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:16:52.443140 26087 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:16:52.443270 26087 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:16:52.443367 26087 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:16:52.476186 26087 rpc_server.cc:307] RPC server started. Bound to: 127.16.217.1:34899
I20260620 20:16:52.476389 26233 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.217.1:34899 every 8 connection(s)
I20260620 20:16:52.477681 26087 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986536687152-17252-0/minicluster-data/ts-0/data/info.pb
I20260620 20:16:52.487258 17252 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu as pid 26087
I20260620 20:16:52.487962 17252 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
/tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986536687152-17252-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986536687152-17252-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986536687152-17252-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986536687152-17252-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.217.2:33487
--local_ip_for_outbound_sockets=127.16.217.2
--tserver_master_addrs=127.16.217.62:44407
--webserver_port=37481
--webserver_interface=127.16.217.2
--keytab_file=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.2.keytab
--principal=kudu/127.16.217.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.217.20:46329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.16.217.2
--keytab_file=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986536687152-17252-0/krb5kdc/oryx_127.16.217.2.keytab with env {KRB5CCNAME=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986536687152-17252-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986536687152-17252-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986536687152-17252-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 20:16:52 dist-test-slave-2hpg krb5kdc[25500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.217.1@KRBTEST.COM for oryx/127.16.217.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 20:16:52 dist-test-slave-2hpg krb5kdc[25500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.217.1@KRBTEST.COM for oryx/127.16.217.62@KRBTEST.COM, Server not found in Kerberos database
W20260620 20:16:52.819900 26238 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:16:52.820207 26238 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:16:52.820370 26238 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:16:52.827527 26238 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:16:52.827687 26238 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:16:52.827863 26238 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.217.2
I20260620 20:16:52.837317 26238 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.217.20:46329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986536687152-17252-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986536687152-17252-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986536687152-17252-0/krb5kdc/oryx_127.16.217.2.keytab
--principal=oryx/127.16.217.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.217.2:33487
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986536687152-17252-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.217.2
--webserver_port=37481
--tserver_master_addrs=127.16.217.62:44407
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.217.2
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986536687152-17252-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:16:52.838733 26238 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:16:52.839995 26238 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:16:52.856042 26245 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:16:52.856357 26244 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:16:52.856412 26247 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:16:52.857748 26238 server_base.cc:1034] running on GCE node
I20260620 20:16:52.858383 26238 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:16:52.859431 26238 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:16:52.860683 26238 hybrid_clock.cc:648] HybridClock initialized: now 1781986612860605 us; error 49 us; skew 500 ppm
I20260620 20:16:52.861069 26238 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:16:52 dist-test-slave-2hpg krb5kdc[25500](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986612, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.217.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:16:52.864543 26238 init.cc:377] Logged in from keytab as oryx/127.16.217.2@KRBTEST.COM (short username oryx)
I20260620 20:16:52.866223 26238 webserver.cc:460] Webserver started at http://127.16.217.2:37481/ using document root <none> and password file <none>
I20260620 20:16:52.866904 26238 fs_manager.cc:359] Metadata directory not provided
I20260620 20:16:52.867043 26238 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:16:52.872298 26238 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.002s
I20260620 20:16:52.875025 26254 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:16:52.876466 26238 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 20:16:52.876636 26238 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986536687152-17252-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986536687152-17252-0/minicluster-data/ts-1/wal
uuid: "b3e9ff6abe074399a453e354224a28fa"
format_stamp: "Formatted at 2026-06-20 20:16:50 on dist-test-slave-2hpg"
server_key: "cc445566fc6fbcf5cd0ac1e654e345f5"
server_key_iv: "bacb0089de00def6f993f000a86fda52"
server_key_version: "encryptionkey@0"
I20260620 20:16:52.877542 26238 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986536687152-17252-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986536687152-17252-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986536687152-17252-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:16:52.894234 26238 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:16:52.898490 26238 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:16:52.898763 26238 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:16:52.899756 26238 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:16:52.901188 26238 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:16:52.901273 26238 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:16:52.901423 26238 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:16:52.901503 26238 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:16:52.940243 26238 rpc_server.cc:307] RPC server started. Bound to: 127.16.217.2:33487
I20260620 20:16:52.940290 26366 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.217.2:33487 every 8 connection(s)
I20260620 20:16:52.941691 26238 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986536687152-17252-0/minicluster-data/ts-1/data/info.pb
I20260620 20:16:52.943333 17252 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu as pid 26238
I20260620 20:16:52.944088 17252 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
/tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986536687152-17252-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986536687152-17252-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986536687152-17252-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986536687152-17252-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.217.3:36363
--local_ip_for_outbound_sockets=127.16.217.3
--tserver_master_addrs=127.16.217.62:44407
--webserver_port=35641
--webserver_interface=127.16.217.3
--keytab_file=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.3.keytab
--principal=kudu/127.16.217.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.217.20:46329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.16.217.3
--keytab_file=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986536687152-17252-0/krb5kdc/oryx_127.16.217.3.keytab with env {KRB5CCNAME=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986536687152-17252-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986536687152-17252-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986536687152-17252-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 20:16:52 dist-test-slave-2hpg krb5kdc[25500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.217.2@KRBTEST.COM for oryx/127.16.217.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 20:16:52 dist-test-slave-2hpg krb5kdc[25500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.217.2@KRBTEST.COM for oryx/127.16.217.62@KRBTEST.COM, Server not found in Kerberos database
W20260620 20:16:53.292230 26371 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:16:53.292548 26371 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:16:53.292728 26371 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:16:53.300045 26371 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:16:53.300200 26371 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:16:53.300395 26371 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.217.3
I20260620 20:16:53.310576 26371 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.217.20:46329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986536687152-17252-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986536687152-17252-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986536687152-17252-0/krb5kdc/oryx_127.16.217.3.keytab
--principal=oryx/127.16.217.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.217.3:36363
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986536687152-17252-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.217.3
--webserver_port=35641
--tserver_master_addrs=127.16.217.62:44407
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.217.3
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986536687152-17252-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:16:53.311976 26371 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:16:53.313354 26371 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:16:53.330492 26377 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:16:53.342308 26378 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:16:53.344339 26381 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:16:53.344812 26371 server_base.cc:1034] running on GCE node
I20260620 20:16:53.345961 26371 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:16:53.350262 26371 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:16:53.351744 26371 hybrid_clock.cc:648] HybridClock initialized: now 1781986613351642 us; error 86 us; skew 500 ppm
I20260620 20:16:53.352326 26371 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:16:53 dist-test-slave-2hpg krb5kdc[25500](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986613, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.217.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:16:53.358402 26371 init.cc:377] Logged in from keytab as oryx/127.16.217.3@KRBTEST.COM (short username oryx)
I20260620 20:16:53.361063 26371 webserver.cc:460] Webserver started at http://127.16.217.3:35641/ using document root <none> and password file <none>
I20260620 20:16:53.362339 26371 fs_manager.cc:359] Metadata directory not provided
I20260620 20:16:53.362545 26371 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:16:53.372721 26371 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.005s sys 0.000s
I20260620 20:16:53.377769 26387 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:16:53.380360 26371 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.000s
I20260620 20:16:53.380609 26371 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986536687152-17252-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986536687152-17252-0/minicluster-data/ts-2/wal
uuid: "eba861f191d54cbb88cb4c9f99f7a605"
format_stamp: "Formatted at 2026-06-20 20:16:50 on dist-test-slave-2hpg"
server_key: "2e371681a81728e83f024df35d9629b7"
server_key_iv: "c973f41ba2e8ee473b5ebd0f56b92587"
server_key_version: "encryptionkey@0"
I20260620 20:16:53.382557 26371 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986536687152-17252-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986536687152-17252-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986536687152-17252-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:16:53.429061 26371 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:16:53.436774 26371 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:16:53.437173 26371 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:16:53.438898 26371 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:16:53.441334 26371 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:16:53.441511 26371 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:16:53.441670 26371 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:16:53.441792 26371 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:16:53.516610 26371 rpc_server.cc:307] RPC server started. Bound to: 127.16.217.3:36363
I20260620 20:16:53.516927 26499 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.217.3:36363 every 8 connection(s)
I20260620 20:16:53.519207 26371 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986536687152-17252-0/minicluster-data/ts-2/data/info.pb
I20260620 20:16:53.527277 17252 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu as pid 26371
Jun 20 20:16:53 dist-test-slave-2hpg krb5kdc[25500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.217.3@KRBTEST.COM for oryx/127.16.217.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 20:16:53 dist-test-slave-2hpg krb5kdc[25500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.217.3@KRBTEST.COM for oryx/127.16.217.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 20:16:53.561728 26503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:53.529043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56106 (local address 127.16.217.62:44407)
0620 20:16:53.538754 (+ 9711us) server_negotiation.cc:206] Beginning negotiation
0620 20:16:53.538765 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 20:16:53.544445 (+ 5680us) server_negotiation.cc:407] Connection header received
0620 20:16:53.544658 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:16:53.544667 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:16:53.544786 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:16:53.544902 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:16:53.547004 (+ 2102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:53.547785 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:53.550432 (+ 2647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:53.550840 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:53.550972 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:53.554357 (+ 3385us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:16:53.554366 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:16:53.554384 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:16:53.554419 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:16:53.557564 (+ 3145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:16:53.558367 (+ 803us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:16:53.558375 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:16:53.558382 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:16:53.558532 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:16:53.558914 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:16:53.558919 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:16:53.558925 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:16:53.559367 (+ 442us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:16:53.559703 (+ 336us) server_negotiation.cc:1035] Waiting for connection context
0620 20:16:53.560096 (+ 393us) server_negotiation.cc:299] Negotiation successful
0620 20:16:53.560554 (+ 458us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9506,"thread_start_us":8966,"threads_started":1}
W20260620 20:16:53.941642 26237 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.450s user 0.006s sys 0.010s
I20260620 20:16:53.944469 26237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:52.480873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:44407 (local address 127.16.217.1:42497)
0620 20:16:52.481672 (+ 799us) negotiation.cc:107] Waiting for socket to connect
0620 20:16:52.481722 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0620 20:16:52.483415 (+ 1693us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:16:52.484779 (+ 1364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:16:52.484786 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:16:52.485285 (+ 499us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:16:52.485812 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:16:52.485829 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:16:52.488104 (+ 2275us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:16:52.488112 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:16:52.489391 (+ 1279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:16:52.489402 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:16:52.490894 (+ 1492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:16:52.490898 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:16:52.491089 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:52.491782 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:16:52.491804 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:16:53.942894 (+1451090us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.217.62:44407: Server oryx/127.16.217.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":526,"thread_start_us":327,"threads_started":1}
W20260620 20:16:53.946121 26236 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:16:52.480765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.217.1:42497 (local address 127.16.217.62:44407)
0620 20:16:52.481456 (+ 691us) server_negotiation.cc:206] Beginning negotiation
0620 20:16:52.481467 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 20:16:52.481923 (+ 456us) server_negotiation.cc:407] Connection header received
0620 20:16:52.483990 (+ 2067us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:16:52.484003 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:16:52.484343 (+ 340us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:16:52.484646 (+ 303us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:16:52.486047 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:52.487889 (+ 1842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:52.489539 (+ 1650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:52.490668 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:52.490860 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:53.945649 (+1454789us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.217.1:42497: BlockingRecv error: recv got EOF from 127.16.217.1:42497 (error 108)
Metrics: {"server-negotiator.queue_time_us":433,"thread_start_us":259,"threads_started":1}
W20260620 20:16:53.946686 26234 heartbeater.cc:644] Failed to heartbeat to 127.16.217.62:44407 (0 consecutive failures): Not authorized: Failed to ping master at 127.16.217.62:44407: Client connection negotiation failed: client connection to 127.16.217.62:44407: Server oryx/127.16.217.62@KRBTEST.COM not found in Kerberos database
Jun 20 20:16:53 dist-test-slave-2hpg krb5kdc[25500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.217.1@KRBTEST.COM for oryx/127.16.217.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 20:16:53 dist-test-slave-2hpg krb5kdc[25500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.217.1@KRBTEST.COM for oryx/127.16.217.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 20:16:53.971378 26237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:53.947781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:44407 (local address 127.16.217.1:60987)
0620 20:16:53.948860 (+ 1079us) negotiation.cc:107] Waiting for socket to connect
0620 20:16:53.948887 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0620 20:16:53.949103 (+ 216us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:16:53.949817 (+ 714us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:16:53.949826 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:16:53.950503 (+ 677us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:16:53.950805 (+ 302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:16:53.950821 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:16:53.953010 (+ 2189us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:16:53.953019 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:16:53.957521 (+ 4502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:16:53.957544 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:16:53.959043 (+ 1499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:16:53.959053 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:16:53.959339 (+ 286us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:53.960533 (+ 1194us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:16:53.960577 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:16:53.970878 (+ 10301us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.217.62:44407: Server oryx/127.16.217.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":909}
W20260620 20:16:53.971632 26236 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:16:53.948413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.217.1:60987 (local address 127.16.217.62:44407)
0620 20:16:53.948651 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0620 20:16:53.948661 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 20:16:53.948973 (+ 312us) server_negotiation.cc:407] Connection header received
0620 20:16:53.949312 (+ 339us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:16:53.949321 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:16:53.949428 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:16:53.949524 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:16:53.951041 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:53.952718 (+ 1677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:53.957747 (+ 5029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:53.958765 (+ 1018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:53.958982 (+ 217us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:53.971248 (+ 12266us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.217.1:60987: BlockingRecv error: recv got EOF from 127.16.217.1:60987 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
Jun 20 20:16:53 dist-test-slave-2hpg krb5kdc[25500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.217.1@KRBTEST.COM for oryx/127.16.217.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 20:16:53 dist-test-slave-2hpg krb5kdc[25500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.217.1@KRBTEST.COM for oryx/127.16.217.62@KRBTEST.COM, Server not found in Kerberos database
W20260620 20:16:53.993391 26236 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:16:53.973595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.217.1:39281 (local address 127.16.217.62:44407)
0620 20:16:53.973826 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0620 20:16:53.973836 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 20:16:53.973950 (+ 114us) server_negotiation.cc:407] Connection header received
0620 20:16:53.974412 (+ 462us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:16:53.974421 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:16:53.974559 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:16:53.974655 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:16:53.976509 (+ 1854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:53.978283 (+ 1774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:53.981087 (+ 2804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:53.982127 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:53.982268 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:53.993006 (+ 10738us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.217.1:39281: BlockingRecv error: recv got EOF from 127.16.217.1:39281 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
I20260620 20:16:53.996232 26507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:53.972928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:44407 (local address 127.16.217.1:39281)
0620 20:16:53.973787 (+ 859us) negotiation.cc:107] Waiting for socket to connect
0620 20:16:53.973816 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0620 20:16:53.974219 (+ 403us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:16:53.974955 (+ 736us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:16:53.974963 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:16:53.975862 (+ 899us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:16:53.976293 (+ 431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:16:53.976310 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:16:53.978492 (+ 2182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:16:53.978501 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:16:53.980838 (+ 2337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:16:53.980858 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:16:53.982659 (+ 1801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:16:53.982668 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:16:53.982972 (+ 304us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:53.986114 (+ 3142us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:16:53.986150 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:16:53.992678 (+ 6528us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.217.62:44407: Server oryx/127.16.217.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":657,"thread_start_us":893,"threads_started":1}
W20260620 20:16:53.996902 26234 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 20:16:54.342195 26369 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.380s user 0.005s sys 0.013s
W20260620 20:16:54.343861 26370 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:16:52.946137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.217.2:54409 (local address 127.16.217.62:44407)
0620 20:16:52.953633 (+ 7496us) server_negotiation.cc:206] Beginning negotiation
0620 20:16:52.953638 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:16:52.953655 (+ 17us) server_negotiation.cc:407] Connection header received
0620 20:16:52.953713 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:16:52.953717 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:16:52.953774 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:16:52.953925 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:16:52.955658 (+ 1733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:52.956944 (+ 1286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:52.959548 (+ 2604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:52.960310 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:52.960435 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:54.343362 (+1382927us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.217.2:54409: BlockingRecv error: recv got EOF from 127.16.217.2:54409 (error 108)
Metrics: {"server-negotiator.queue_time_us":7389,"thread_start_us":7281,"threads_started":1}
I20260620 20:16:54.344002 26369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:52.945579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:44407 (local address 127.16.217.2:54409)
0620 20:16:52.946771 (+ 1192us) negotiation.cc:107] Waiting for socket to connect
0620 20:16:52.946842 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0620 20:16:52.950539 (+ 3697us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:16:52.954213 (+ 3674us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:16:52.954223 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:16:52.954761 (+ 538us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:16:52.955448 (+ 687us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:16:52.955477 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:16:52.957122 (+ 1645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:16:52.957130 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:16:52.959344 (+ 2214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:16:52.959361 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:16:52.960748 (+ 1387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:16:52.960758 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:16:52.961353 (+ 595us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:52.962347 (+ 994us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:16:52.962382 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:16:54.342981 (+1380599us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.217.62:44407: Server oryx/127.16.217.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":659,"spinlock_wait_cycles":38784,"thread_start_us":583,"threads_started":1}
W20260620 20:16:54.345134 26367 heartbeater.cc:644] Failed to heartbeat to 127.16.217.62:44407 (0 consecutive failures): Not authorized: Failed to ping master at 127.16.217.62:44407: Client connection negotiation failed: client connection to 127.16.217.62:44407: Server oryx/127.16.217.62@KRBTEST.COM not found in Kerberos database
Jun 20 20:16:54 dist-test-slave-2hpg krb5kdc[25500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.217.2@KRBTEST.COM for oryx/127.16.217.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 20:16:54 dist-test-slave-2hpg krb5kdc[25500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.217.2@KRBTEST.COM for oryx/127.16.217.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 20:16:54.361500 26369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:54.346023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:44407 (local address 127.16.217.2:35357)
0620 20:16:54.346297 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0620 20:16:54.346324 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0620 20:16:54.346514 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:16:54.346866 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:16:54.346874 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:16:54.347541 (+ 667us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:16:54.347835 (+ 294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:16:54.347849 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:16:54.348892 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:16:54.348900 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:16:54.351240 (+ 2340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:16:54.351262 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:16:54.352062 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:16:54.352071 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:16:54.352357 (+ 286us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:54.357063 (+ 4706us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:16:54.357082 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:16:54.361180 (+ 4098us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.217.62:44407: Server oryx/127.16.217.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":82}
W20260620 20:16:54.361630 26370 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:16:54.346080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.217.2:35357 (local address 127.16.217.62:44407)
0620 20:16:54.346273 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0620 20:16:54.346278 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:16:54.346422 (+ 144us) server_negotiation.cc:407] Connection header received
0620 20:16:54.346650 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:16:54.346654 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:16:54.346716 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:16:54.346788 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:16:54.348006 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:54.348741 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:54.351439 (+ 2698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:54.351848 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:54.351949 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:54.361379 (+ 9430us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.217.2:35357: BlockingRecv error: recv got EOF from 127.16.217.2:35357 (error 108)
Metrics: {"server-negotiator.queue_time_us":100}
Jun 20 20:16:54 dist-test-slave-2hpg krb5kdc[25500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.217.2@KRBTEST.COM for oryx/127.16.217.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 20:16:54 dist-test-slave-2hpg krb5kdc[25500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.217.2@KRBTEST.COM for oryx/127.16.217.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 20:16:54.374296 26369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:54.362366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:44407 (local address 127.16.217.2:51365)
0620 20:16:54.362834 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0620 20:16:54.362858 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 20:16:54.363022 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:16:54.363412 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:16:54.363420 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:16:54.363986 (+ 566us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:16:54.364271 (+ 285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:16:54.364286 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:16:54.365298 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:16:54.365306 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:16:54.367768 (+ 2462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:16:54.367790 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:16:54.368502 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:16:54.368510 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:16:54.368800 (+ 290us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:54.369499 (+ 699us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:16:54.369532 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:16:54.374085 (+ 4553us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.217.62:44407: Server oryx/127.16.217.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":305}
W20260620 20:16:54.374750 26367 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 20:16:54.374944 26370 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:16:54.362574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.217.2:51365 (local address 127.16.217.62:44407)
0620 20:16:54.362757 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0620 20:16:54.362761 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:16:54.362944 (+ 183us) server_negotiation.cc:407] Connection header received
0620 20:16:54.363155 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:16:54.363158 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:16:54.363240 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:16:54.363313 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:16:54.364462 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:54.365136 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:54.367945 (+ 2809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:54.368335 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:54.368425 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:54.374475 (+ 6050us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.217.2:51365: BlockingRecv error: recv got EOF from 127.16.217.2:51365 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
W20260620 20:16:54.676972 26502 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.125s user 0.007s sys 0.007s
W20260620 20:16:54.678150 26505 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:16:53.528438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.217.3:47407 (local address 127.16.217.62:44407)
0620 20:16:53.538333 (+ 9895us) server_negotiation.cc:206] Beginning negotiation
0620 20:16:53.538343 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 20:16:53.538372 (+ 29us) server_negotiation.cc:407] Connection header received
0620 20:16:53.538457 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:16:53.538470 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:16:53.538582 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:16:53.538736 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:16:53.542204 (+ 3468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:53.544096 (+ 1892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:53.548045 (+ 3949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:53.549090 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:53.549268 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:54.677920 (+1128652us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.217.3:47407: BlockingRecv error: recv got EOF from 127.16.217.3:47407 (error 108)
Metrics: {"server-negotiator.queue_time_us":9701,"thread_start_us":9947,"threads_started":1}
I20260620 20:16:54.678764 26502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:53.525818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:44407 (local address 127.16.217.3:47407)
0620 20:16:53.534518 (+ 8700us) negotiation.cc:107] Waiting for socket to connect
0620 20:16:53.534571 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0620 20:16:53.537763 (+ 3192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:16:53.539281 (+ 1518us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:16:53.539302 (+ 21us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:16:53.540504 (+ 1202us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:16:53.541809 (+ 1305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:16:53.541848 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:16:53.544466 (+ 2618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:16:53.544476 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:16:53.547766 (+ 3290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:16:53.547792 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:16:53.549828 (+ 2036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:16:53.549840 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:16:53.550380 (+ 540us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:53.551577 (+ 1197us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:16:53.551610 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:16:54.677640 (+1126030us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.217.62:44407: Server oryx/127.16.217.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":8306,"thread_start_us":8011,"threads_started":1}
W20260620 20:16:54.679800 26500 heartbeater.cc:644] Failed to heartbeat to 127.16.217.62:44407 (0 consecutive failures): Not authorized: Failed to ping master at 127.16.217.62:44407: Client connection negotiation failed: client connection to 127.16.217.62:44407: Server oryx/127.16.217.62@KRBTEST.COM not found in Kerberos database
Jun 20 20:16:54 dist-test-slave-2hpg krb5kdc[25500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.217.3@KRBTEST.COM for oryx/127.16.217.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 20:16:54 dist-test-slave-2hpg krb5kdc[25500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.217.3@KRBTEST.COM for oryx/127.16.217.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 20:16:54.692838 26502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:54.680430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:44407 (local address 127.16.217.3:50191)
0620 20:16:54.680633 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0620 20:16:54.680650 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 20:16:54.680746 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:16:54.681048 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:16:54.681051 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:16:54.681368 (+ 317us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:16:54.681536 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:16:54.681544 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:16:54.682558 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:16:54.682562 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:16:54.683491 (+ 929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:16:54.683501 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:16:54.684212 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:16:54.684217 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:16:54.684340 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:54.688577 (+ 4237us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:16:54.688595 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:16:54.692546 (+ 3951us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.217.62:44407: Server oryx/127.16.217.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":64}
W20260620 20:16:54.692970 26505 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:16:54.680462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.217.3:50191 (local address 127.16.217.62:44407)
0620 20:16:54.680646 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0620 20:16:54.680650 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:16:54.680722 (+ 72us) server_negotiation.cc:407] Connection header received
0620 20:16:54.680821 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:16:54.680825 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:16:54.680929 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:16:54.681010 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:16:54.681630 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:54.682460 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:54.683639 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:54.684064 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:54.684173 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:54.692760 (+ 8587us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.217.3:50191: BlockingRecv error: recv got EOF from 127.16.217.3:50191 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Jun 20 20:16:54 dist-test-slave-2hpg krb5kdc[25500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.217.3@KRBTEST.COM for oryx/127.16.217.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 20:16:54 dist-test-slave-2hpg krb5kdc[25500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.217.3@KRBTEST.COM for oryx/127.16.217.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 20:16:54.702119 26502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:54.693537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:44407 (local address 127.16.217.3:43413)
0620 20:16:54.693700 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0620 20:16:54.693717 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 20:16:54.693840 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:16:54.694285 (+ 445us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:16:54.694290 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:16:54.694623 (+ 333us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:16:54.694801 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:16:54.694810 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:16:54.695877 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:16:54.695883 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:16:54.696886 (+ 1003us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:16:54.696898 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:16:54.697708 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:16:54.697715 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:16:54.697985 (+ 270us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:54.698549 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:16:54.698566 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:16:54.701820 (+ 3254us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.217.62:44407: Server oryx/127.16.217.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":38}
W20260620 20:16:54.702447 26505 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:16:54.693655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.217.3:43413 (local address 127.16.217.62:44407)
0620 20:16:54.693980 (+ 325us) server_negotiation.cc:206] Beginning negotiation
0620 20:16:54.693987 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:16:54.694004 (+ 17us) server_negotiation.cc:407] Connection header received
0620 20:16:54.694060 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:16:54.694064 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:16:54.694119 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:16:54.694250 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:16:54.694950 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:54.695722 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:54.697041 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:54.697535 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:54.697685 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:54.702172 (+ 4487us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.217.3:43413: BlockingRecv error: recv got EOF from 127.16.217.3:43413 (error 108)
Metrics: {"server-negotiator.queue_time_us":159}
W20260620 20:16:54.702452 26500 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Jun 20 20:16:55 dist-test-slave-2hpg krb5kdc[25500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.217.1@KRBTEST.COM for oryx/127.16.217.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 20:16:55 dist-test-slave-2hpg krb5kdc[25500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.217.1@KRBTEST.COM for oryx/127.16.217.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 20:16:55.007176 26508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:54.997490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:44407 (local address 127.16.217.1:37445)
0620 20:16:54.998156 (+ 666us) negotiation.cc:107] Waiting for socket to connect
0620 20:16:54.998187 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0620 20:16:54.998353 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:16:54.998733 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:16:54.998741 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:16:54.999143 (+ 402us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:16:54.999289 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:16:54.999297 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:16:55.000268 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:16:55.000274 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:16:55.001379 (+ 1105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:16:55.001391 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:16:55.002301 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:16:55.002308 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:16:55.002488 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:55.003096 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:16:55.003112 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:16:55.006952 (+ 3840us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.217.62:44407: Server oryx/127.16.217.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":533,"thread_start_us":412,"threads_started":1}
W20260620 20:16:55.007637 26505 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:16:54.997560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.217.1:37445 (local address 127.16.217.62:44407)
0620 20:16:54.997767 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0620 20:16:54.997777 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 20:16:54.998282 (+ 505us) server_negotiation.cc:407] Connection header received
0620 20:16:54.998482 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:16:54.998487 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:16:54.998587 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:16:54.998689 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:16:54.999437 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:55.000117 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:55.001532 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:55.002110 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:55.002247 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:55.007165 (+ 4918us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.217.1:37445: BlockingRecv error: recv got EOF from 127.16.217.1:37445 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
Jun 20 20:16:55 dist-test-slave-2hpg krb5kdc[25500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.217.2@KRBTEST.COM for oryx/127.16.217.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 20:16:55 dist-test-slave-2hpg krb5kdc[25500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.217.2@KRBTEST.COM for oryx/127.16.217.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 20:16:55.384142 26509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:55.375344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:44407 (local address 127.16.217.2:43499)
0620 20:16:55.375918 (+ 574us) negotiation.cc:107] Waiting for socket to connect
0620 20:16:55.375933 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 20:16:55.376044 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:16:55.376349 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:16:55.376352 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:16:55.376767 (+ 415us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:16:55.376906 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:16:55.376914 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:16:55.377971 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:16:55.377976 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:16:55.378947 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:16:55.378958 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:16:55.379649 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:16:55.379653 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:16:55.379780 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:55.380328 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:16:55.380346 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:16:55.383909 (+ 3563us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.217.62:44407: Server oryx/127.16.217.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":353,"threads_started":1}
W20260620 20:16:55.384363 26505 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:16:55.375383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.217.2:43499 (local address 127.16.217.62:44407)
0620 20:16:55.375538 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0620 20:16:55.375543 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:16:55.376003 (+ 460us) server_negotiation.cc:407] Connection header received
0620 20:16:55.376153 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:16:55.376157 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:16:55.376223 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:16:55.376299 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:16:55.377022 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:55.377769 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:55.379089 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:55.379497 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:55.379591 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:55.384115 (+ 4524us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.217.2:43499: BlockingRecv error: recv got EOF from 127.16.217.2:43499 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Jun 20 20:16:55 dist-test-slave-2hpg krb5kdc[25500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.217.3@KRBTEST.COM for oryx/127.16.217.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 20:16:55 dist-test-slave-2hpg krb5kdc[25500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.217.3@KRBTEST.COM for oryx/127.16.217.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 20:16:55.711987 26510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:55.702975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:44407 (local address 127.16.217.3:55691)
0620 20:16:55.703488 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0620 20:16:55.703503 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 20:16:55.703651 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:16:55.704005 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:16:55.704010 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:16:55.704438 (+ 428us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:16:55.704574 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:16:55.704582 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:16:55.705590 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:16:55.705597 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:16:55.706674 (+ 1077us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:16:55.706686 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:16:55.707423 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:16:55.707429 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:16:55.707594 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:55.708221 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:16:55.708238 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:16:55.711767 (+ 3529us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.217.62:44407: Server oryx/127.16.217.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":426,"thread_start_us":308,"threads_started":1}
W20260620 20:16:55.712159 26505 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:16:55.703048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.217.3:55691 (local address 127.16.217.62:44407)
0620 20:16:55.703264 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0620 20:16:55.703272 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 20:16:55.703582 (+ 310us) server_negotiation.cc:407] Connection header received
0620 20:16:55.703788 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:16:55.703795 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:16:55.703873 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:16:55.703973 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:16:55.704723 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:55.705456 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:55.706815 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:55.707250 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:55.707391 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:55.711943 (+ 4552us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.217.3:55691: BlockingRecv error: recv got EOF from 127.16.217.3:55691 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Jun 20 20:16:56 dist-test-slave-2hpg krb5kdc[25500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.217.1@KRBTEST.COM for oryx/127.16.217.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 20:16:56 dist-test-slave-2hpg krb5kdc[25500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.217.1@KRBTEST.COM for oryx/127.16.217.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 20:16:56.016903 26511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:56.008151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:44407 (local address 127.16.217.1:52713)
0620 20:16:56.008602 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0620 20:16:56.008616 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 20:16:56.008781 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:16:56.009105 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:16:56.009110 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:16:56.009480 (+ 370us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:16:56.009620 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:16:56.009628 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:16:56.010689 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:16:56.010694 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:16:56.011637 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:16:56.011648 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:16:56.012343 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:16:56.012349 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:16:56.012525 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:56.013127 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:16:56.013143 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:16:56.016565 (+ 3422us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.217.62:44407: Server oryx/127.16.217.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":238,"threads_started":1}
W20260620 20:16:56.017055 26505 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:16:56.008178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.217.1:52713 (local address 127.16.217.62:44407)
0620 20:16:56.008419 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0620 20:16:56.008428 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 20:16:56.008696 (+ 268us) server_negotiation.cc:407] Connection header received
0620 20:16:56.008915 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:16:56.008919 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:16:56.008979 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:16:56.009074 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:16:56.009798 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:56.010558 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:56.011783 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:56.012199 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:56.012340 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:56.016800 (+ 4460us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.217.1:52713: BlockingRecv error: recv got EOF from 127.16.217.1:52713 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
Jun 20 20:16:56 dist-test-slave-2hpg krb5kdc[25500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.217.2@KRBTEST.COM for oryx/127.16.217.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 20:16:56 dist-test-slave-2hpg krb5kdc[25500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.217.2@KRBTEST.COM for oryx/127.16.217.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 20:16:56.394966 26512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:56.385080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:44407 (local address 127.16.217.2:56483)
0620 20:16:56.385688 (+ 608us) negotiation.cc:107] Waiting for socket to connect
0620 20:16:56.385705 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 20:16:56.385919 (+ 214us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:16:56.386320 (+ 401us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:16:56.386324 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:16:56.386785 (+ 461us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:16:56.386965 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:16:56.386997 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:16:56.388045 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:16:56.388049 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:16:56.389026 (+ 977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:16:56.389038 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:16:56.389792 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:16:56.389796 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:16:56.389972 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:56.390586 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:16:56.390607 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:16:56.394641 (+ 4034us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.217.62:44407: Server oryx/127.16.217.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":491,"thread_start_us":363,"threads_started":1}
W20260620 20:16:56.395064 26505 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:16:56.385219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.217.2:56483 (local address 127.16.217.62:44407)
0620 20:16:56.385461 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0620 20:16:56.385468 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:16:56.385795 (+ 327us) server_negotiation.cc:407] Connection header received
0620 20:16:56.386057 (+ 262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:16:56.386067 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:16:56.386150 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:16:56.386252 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:16:56.387144 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:56.387911 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:56.389148 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:56.389635 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:56.389793 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:56.394853 (+ 5060us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.217.2:56483: BlockingRecv error: recv got EOF from 127.16.217.2:56483 (error 108)
Metrics: {"server-negotiator.queue_time_us":87}
Jun 20 20:16:56 dist-test-slave-2hpg krb5kdc[25500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.217.3@KRBTEST.COM for oryx/127.16.217.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 20:16:56 dist-test-slave-2hpg krb5kdc[25500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.217.3@KRBTEST.COM for oryx/127.16.217.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 20:16:56.721906 26513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:56.712878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:44407 (local address 127.16.217.3:55665)
0620 20:16:56.713411 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0620 20:16:56.713426 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 20:16:56.713581 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:16:56.713998 (+ 417us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:16:56.714003 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:16:56.714443 (+ 440us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:16:56.714624 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:16:56.714634 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:16:56.715700 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:16:56.715704 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:16:56.716610 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:16:56.716622 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:16:56.717328 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:16:56.717334 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:16:56.717516 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:56.718181 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:16:56.718202 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:16:56.721485 (+ 3283us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.217.62:44407: Server oryx/127.16.217.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":322,"threads_started":1}
W20260620 20:16:56.721946 26505 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:16:56.713100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.217.3:55665 (local address 127.16.217.62:44407)
0620 20:16:56.713331 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0620 20:16:56.713337 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:16:56.713512 (+ 175us) server_negotiation.cc:407] Connection header received
0620 20:16:56.713746 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:16:56.713751 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:16:56.713839 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:16:56.713965 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:16:56.714778 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:56.715525 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:56.716760 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:56.717176 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:56.717311 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:56.721678 (+ 4367us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.217.3:55665: BlockingRecv error: recv got EOF from 127.16.217.3:55665 (error 108)
Metrics: {"server-negotiator.queue_time_us":98}
Jun 20 20:16:57 dist-test-slave-2hpg krb5kdc[25500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.217.1@KRBTEST.COM for oryx/127.16.217.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 20:16:57 dist-test-slave-2hpg krb5kdc[25500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.217.1@KRBTEST.COM for oryx/127.16.217.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 20:16:57.027463 26514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:57.018065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:44407 (local address 127.16.217.1:58397)
0620 20:16:57.018608 (+ 543us) negotiation.cc:107] Waiting for socket to connect
0620 20:16:57.018624 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 20:16:57.018737 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:16:57.019023 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:16:57.019026 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:16:57.019402 (+ 376us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:16:57.019554 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:16:57.019562 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:16:57.020651 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:16:57.020655 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:16:57.021621 (+ 966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:16:57.021632 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:16:57.022515 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:16:57.022520 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:16:57.022672 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:57.023278 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:16:57.023295 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:16:57.027229 (+ 3934us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.217.62:44407: Server oryx/127.16.217.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":417,"thread_start_us":318,"threads_started":1}
W20260620 20:16:57.027637 26505 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:16:57.018126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.217.1:58397 (local address 127.16.217.62:44407)
0620 20:16:57.018328 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0620 20:16:57.018333 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:16:57.018698 (+ 365us) server_negotiation.cc:407] Connection header received
0620 20:16:57.018833 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:16:57.018837 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:16:57.018897 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:16:57.018979 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:16:57.019717 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:57.020508 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:57.021768 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:57.022350 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:57.022461 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:57.027417 (+ 4956us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.217.1:58397: BlockingRecv error: recv got EOF from 127.16.217.1:58397 (error 108)
Metrics: {"server-negotiator.queue_time_us":88}
Jun 20 20:16:57 dist-test-slave-2hpg krb5kdc[25500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.217.2@KRBTEST.COM for oryx/127.16.217.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 20:16:57 dist-test-slave-2hpg krb5kdc[25500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.217.2@KRBTEST.COM for oryx/127.16.217.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 20:16:57.404217 26515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:57.395954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:44407 (local address 127.16.217.2:43437)
0620 20:16:57.396597 (+ 643us) negotiation.cc:107] Waiting for socket to connect
0620 20:16:57.396613 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 20:16:57.396727 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:16:57.396998 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:16:57.397001 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:16:57.397356 (+ 355us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:16:57.397514 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:16:57.397522 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:16:57.398463 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:16:57.398467 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:16:57.399373 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:16:57.399383 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:16:57.400041 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:16:57.400045 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:16:57.400161 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:57.400738 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:16:57.400761 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:16:57.403987 (+ 3226us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.217.62:44407: Server oryx/127.16.217.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":540,"thread_start_us":404,"threads_started":1}
W20260620 20:16:57.404407 26505 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:16:57.396001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.217.2:43437 (local address 127.16.217.62:44407)
0620 20:16:57.396226 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0620 20:16:57.396235 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 20:16:57.396692 (+ 457us) server_negotiation.cc:407] Connection header received
0620 20:16:57.396824 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:16:57.396828 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:16:57.396886 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:16:57.396956 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:16:57.397630 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:57.398355 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:57.399482 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:57.399923 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:57.400019 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:57.404164 (+ 4145us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.217.2:43437: BlockingRecv error: recv got EOF from 127.16.217.2:43437 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Jun 20 20:16:57 dist-test-slave-2hpg krb5kdc[25500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.217.3@KRBTEST.COM for oryx/127.16.217.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 20:16:57 dist-test-slave-2hpg krb5kdc[25500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.217.3@KRBTEST.COM for oryx/127.16.217.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 20:16:57.732028 26516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:57.723178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:44407 (local address 127.16.217.3:44249)
0620 20:16:57.723685 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0620 20:16:57.723705 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 20:16:57.723876 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:16:57.724226 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:16:57.724230 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:16:57.724714 (+ 484us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:16:57.724878 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:16:57.724887 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:16:57.725938 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:16:57.725943 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:16:57.726851 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:16:57.726861 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:16:57.727649 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:16:57.727655 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:16:57.727826 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:57.728446 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:16:57.728468 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:16:57.731797 (+ 3329us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.217.62:44407: Server oryx/127.16.217.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":415,"thread_start_us":253,"threads_started":1}
W20260620 20:16:57.732295 26505 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:16:57.723232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.217.3:44249 (local address 127.16.217.62:44407)
0620 20:16:57.723497 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0620 20:16:57.723503 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:16:57.723788 (+ 285us) server_negotiation.cc:407] Connection header received
0620 20:16:57.723994 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:16:57.724000 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:16:57.724078 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:16:57.724180 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:16:57.725033 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:57.725766 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:57.726992 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:57.727512 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:57.727640 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:57.731962 (+ 4322us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.217.3:44249: BlockingRecv error: recv got EOF from 127.16.217.3:44249 (error 108)
Metrics: {"server-negotiator.queue_time_us":100}
Jun 20 20:16:58 dist-test-slave-2hpg krb5kdc[25500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.217.1@KRBTEST.COM for oryx/127.16.217.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 20:16:58 dist-test-slave-2hpg krb5kdc[25500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.217.1@KRBTEST.COM for oryx/127.16.217.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 20:16:58.037750 26517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:58.028400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:44407 (local address 127.16.217.1:33145)
0620 20:16:58.028914 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0620 20:16:58.028931 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 20:16:58.029087 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:16:58.029484 (+ 397us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:16:58.029489 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:16:58.029917 (+ 428us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:16:58.030069 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:16:58.030077 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:16:58.031080 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:16:58.031088 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:16:58.032085 (+ 997us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:16:58.032095 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:16:58.032777 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:16:58.032783 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:16:58.032951 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:58.033581 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:16:58.033598 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:16:58.037382 (+ 3784us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.217.62:44407: Server oryx/127.16.217.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":425,"thread_start_us":306,"threads_started":1}
W20260620 20:16:58.037897 26505 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:16:58.028474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.217.1:33145 (local address 127.16.217.62:44407)
0620 20:16:58.028704 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0620 20:16:58.028712 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 20:16:58.029017 (+ 305us) server_negotiation.cc:407] Connection header received
0620 20:16:58.029246 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:16:58.029250 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:16:58.029332 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:16:58.029439 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:16:58.030216 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:58.030937 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:58.032234 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:58.032632 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:58.032756 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:58.037593 (+ 4837us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.217.1:33145: BlockingRecv error: recv got EOF from 127.16.217.1:33145 (error 108)
Metrics: {"server-negotiator.queue_time_us":89}
Jun 20 20:16:58 dist-test-slave-2hpg krb5kdc[25500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.217.2@KRBTEST.COM for oryx/127.16.217.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 20:16:58 dist-test-slave-2hpg krb5kdc[25500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.217.2@KRBTEST.COM for oryx/127.16.217.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 20:16:58.414227 26518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:58.405322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:44407 (local address 127.16.217.2:34383)
0620 20:16:58.405905 (+ 583us) negotiation.cc:107] Waiting for socket to connect
0620 20:16:58.405920 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 20:16:58.406100 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:16:58.406452 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:16:58.406456 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:16:58.406844 (+ 388us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:16:58.407002 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:16:58.407011 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:16:58.408025 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:16:58.408031 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:16:58.409037 (+ 1006us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:16:58.409048 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:16:58.409783 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:16:58.409788 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:16:58.410006 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:58.410599 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:16:58.410616 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:16:58.413972 (+ 3356us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.217.62:44407: Server oryx/127.16.217.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":314,"threads_started":1}
W20260620 20:16:58.414352 26505 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:16:58.405421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.217.2:34383 (local address 127.16.217.62:44407)
0620 20:16:58.405688 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0620 20:16:58.405696 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 20:16:58.406015 (+ 319us) server_negotiation.cc:407] Connection header received
0620 20:16:58.406236 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:16:58.406243 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:16:58.406319 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:16:58.406422 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:16:58.407149 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:58.407884 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:58.409181 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:58.409619 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:58.409770 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:58.414166 (+ 4396us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.217.2:34383: BlockingRecv error: recv got EOF from 127.16.217.2:34383 (error 108)
Metrics: {"server-negotiator.queue_time_us":101}
Jun 20 20:16:58 dist-test-slave-2hpg krb5kdc[25500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.217.3@KRBTEST.COM for oryx/127.16.217.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 20:16:58 dist-test-slave-2hpg krb5kdc[25500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.217.3@KRBTEST.COM for oryx/127.16.217.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 20:16:58.742110 26519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:58.732979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:44407 (local address 127.16.217.3:50919)
0620 20:16:58.733568 (+ 589us) negotiation.cc:107] Waiting for socket to connect
0620 20:16:58.733590 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 20:16:58.733737 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:16:58.734185 (+ 448us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:16:58.734188 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:16:58.734549 (+ 361us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:16:58.734727 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:16:58.734738 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:16:58.735799 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:16:58.735804 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:16:58.736828 (+ 1024us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:16:58.736841 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:16:58.737471 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:16:58.737476 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:16:58.737614 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:58.738314 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:16:58.738331 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:16:58.741751 (+ 3420us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.217.62:44407: Server oryx/127.16.217.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":469,"thread_start_us":571,"threads_started":1}
W20260620 20:16:58.742244 26505 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:16:58.733145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.217.3:50919 (local address 127.16.217.62:44407)
0620 20:16:58.733333 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0620 20:16:58.733337 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:16:58.733707 (+ 370us) server_negotiation.cc:407] Connection header received
0620 20:16:58.733910 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:16:58.733918 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:16:58.734001 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:16:58.734098 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:16:58.734901 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:58.735646 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:58.736951 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:58.737328 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:58.737431 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:58.741971 (+ 4540us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.217.3:50919: BlockingRecv error: recv got EOF from 127.16.217.3:50919 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Jun 20 20:16:59 dist-test-slave-2hpg krb5kdc[25500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.217.1@KRBTEST.COM for oryx/127.16.217.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 20:16:59 dist-test-slave-2hpg krb5kdc[25500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.217.1@KRBTEST.COM for oryx/127.16.217.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 20:16:59.047741 26520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:59.038753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:44407 (local address 127.16.217.1:35245)
0620 20:16:59.039217 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0620 20:16:59.039235 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 20:16:59.039361 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:16:59.039738 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:16:59.039742 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:16:59.040132 (+ 390us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:16:59.040283 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:16:59.040291 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:16:59.041248 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:16:59.041253 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:16:59.042326 (+ 1073us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:16:59.042337 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:16:59.043232 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:16:59.043241 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:16:59.043377 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:59.044055 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:16:59.044072 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:16:59.047523 (+ 3451us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.217.62:44407: Server oryx/127.16.217.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":259,"threads_started":1}
W20260620 20:16:59.048010 26505 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:16:59.038775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.217.1:35245 (local address 127.16.217.62:44407)
0620 20:16:59.038995 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0620 20:16:59.039005 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 20:16:59.039328 (+ 323us) server_negotiation.cc:407] Connection header received
0620 20:16:59.039458 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:16:59.039461 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:16:59.039521 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:16:59.039609 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:16:59.040409 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:59.041114 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:59.042467 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:59.043046 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:59.043171 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:59.047718 (+ 4547us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.217.1:35245: BlockingRecv error: recv got EOF from 127.16.217.1:35245 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Jun 20 20:16:59 dist-test-slave-2hpg krb5kdc[25500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.217.2@KRBTEST.COM for oryx/127.16.217.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 20:16:59 dist-test-slave-2hpg krb5kdc[25500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.217.2@KRBTEST.COM for oryx/127.16.217.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 20:16:59.428990 26521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:59.415133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:44407 (local address 127.16.217.2:32995)
0620 20:16:59.415661 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0620 20:16:59.415684 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 20:16:59.415803 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:16:59.416136 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:16:59.416140 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:16:59.416558 (+ 418us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:16:59.416748 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:16:59.416757 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:16:59.417798 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:16:59.417803 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:16:59.418847 (+ 1044us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:16:59.418859 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:16:59.419539 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:16:59.419543 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:16:59.419660 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:59.425026 (+ 5366us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:16:59.425047 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:16:59.428749 (+ 3702us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.217.62:44407: Server oryx/127.16.217.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":411,"thread_start_us":282,"threads_started":1}
W20260620 20:16:59.429210 26505 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:16:59.415350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.217.2:32995 (local address 127.16.217.62:44407)
0620 20:16:59.415517 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0620 20:16:59.415521 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:16:59.415775 (+ 254us) server_negotiation.cc:407] Connection header received
0620 20:16:59.415902 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:16:59.415906 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:16:59.415967 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:16:59.416053 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:16:59.416887 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:59.417620 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:59.418973 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:59.419405 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:59.419502 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:59.428947 (+ 9445us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.217.2:32995: BlockingRecv error: recv got EOF from 127.16.217.2:32995 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Jun 20 20:16:59 dist-test-slave-2hpg krb5kdc[25500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.217.3@KRBTEST.COM for oryx/127.16.217.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 20:16:59 dist-test-slave-2hpg krb5kdc[25500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.217.3@KRBTEST.COM for oryx/127.16.217.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 20:16:59.752492 26522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:59.743191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:44407 (local address 127.16.217.3:40323)
0620 20:16:59.743834 (+ 643us) negotiation.cc:107] Waiting for socket to connect
0620 20:16:59.743850 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 20:16:59.744014 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:16:59.744406 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:16:59.744412 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:16:59.744872 (+ 460us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:16:59.745017 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:16:59.745026 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:16:59.746113 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:16:59.746122 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:16:59.747050 (+ 928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:16:59.747061 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:16:59.747828 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:16:59.747834 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:16:59.747994 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:59.748710 (+ 716us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:16:59.748737 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:16:59.752163 (+ 3426us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.217.62:44407: Server oryx/127.16.217.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":553,"thread_start_us":353,"threads_started":1}
W20260620 20:16:59.752590 26505 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:16:59.743256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.217.3:40323 (local address 127.16.217.62:44407)
0620 20:16:59.743597 (+ 341us) server_negotiation.cc:206] Beginning negotiation
0620 20:16:59.743606 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 20:16:59.743947 (+ 341us) server_negotiation.cc:407] Connection header received
0620 20:16:59.744147 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:16:59.744151 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:16:59.744244 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:16:59.744349 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:16:59.745156 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:59.745974 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:59.747192 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:59.747681 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:59.747810 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:59.752374 (+ 4564us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.217.3:40323: BlockingRecv error: recv got EOF from 127.16.217.3:40323 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
Jun 20 20:17:00 dist-test-slave-2hpg krb5kdc[25500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.217.1@KRBTEST.COM for oryx/127.16.217.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 20:17:00 dist-test-slave-2hpg krb5kdc[25500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.217.1@KRBTEST.COM for oryx/127.16.217.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 20:17:00.058032 26523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:00.048719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:44407 (local address 127.16.217.1:45827)
0620 20:17:00.049216 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0620 20:17:00.049233 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 20:17:00.049361 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:17:00.049653 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:17:00.049656 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:17:00.050278 (+ 622us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:17:00.050479 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:17:00.050488 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:17:00.051729 (+ 1241us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:17:00.051734 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:17:00.052646 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:17:00.052656 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:17:00.053398 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:17:00.053404 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:17:00.053577 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:17:00.054210 (+ 633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:17:00.054227 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:17:00.057712 (+ 3485us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.217.62:44407: Server oryx/127.16.217.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":386,"thread_start_us":281,"threads_started":1}
W20260620 20:17:00.058272 26505 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:17:00.048773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.217.1:45827 (local address 127.16.217.62:44407)
0620 20:17:00.049016 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0620 20:17:00.049026 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 20:17:00.049314 (+ 288us) server_negotiation.cc:407] Connection header received
0620 20:17:00.049458 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:17:00.049461 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:17:00.049522 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:17:00.049600 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:17:00.050624 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:00.051594 (+ 970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:00.052802 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:00.053244 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:00.053385 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:17:00.058002 (+ 4617us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.217.1:45827: BlockingRecv error: recv got EOF from 127.16.217.1:45827 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Jun 20 20:17:00 dist-test-slave-2hpg krb5kdc[25500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.217.2@KRBTEST.COM for oryx/127.16.217.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 20:17:00 dist-test-slave-2hpg krb5kdc[25500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.217.2@KRBTEST.COM for oryx/127.16.217.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 20:17:00.439453 26524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:00.430114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:44407 (local address 127.16.217.2:53259)
0620 20:17:00.430689 (+ 575us) negotiation.cc:107] Waiting for socket to connect
0620 20:17:00.430705 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 20:17:00.430823 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:17:00.431163 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:17:00.431167 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:17:00.431549 (+ 382us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:17:00.431738 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:17:00.431748 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:17:00.433046 (+ 1298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:17:00.433051 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:17:00.434241 (+ 1190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:17:00.434254 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:17:00.435007 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:17:00.435012 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:17:00.435129 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:17:00.435766 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:17:00.435783 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:17:00.439231 (+ 3448us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.217.62:44407: Server oryx/127.16.217.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":341,"threads_started":1}
W20260620 20:17:00.439709 26505 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:17:00.430238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.217.2:53259 (local address 127.16.217.62:44407)
0620 20:17:00.430409 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0620 20:17:00.430413 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:17:00.430815 (+ 402us) server_negotiation.cc:407] Connection header received
0620 20:17:00.430930 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:17:00.430934 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:17:00.431023 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:17:00.431123 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:17:00.431899 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:00.432860 (+ 961us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:00.434375 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:00.434860 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:00.434967 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:17:00.439427 (+ 4460us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.217.2:53259: BlockingRecv error: recv got EOF from 127.16.217.2:53259 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Jun 20 20:17:00 dist-test-slave-2hpg krb5kdc[25500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.217.3@KRBTEST.COM for oryx/127.16.217.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 20:17:00 dist-test-slave-2hpg krb5kdc[25500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.217.3@KRBTEST.COM for oryx/127.16.217.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 20:17:00.762317 26525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:00.753617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:44407 (local address 127.16.217.3:40055)
0620 20:17:00.754191 (+ 574us) negotiation.cc:107] Waiting for socket to connect
0620 20:17:00.754211 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 20:17:00.754343 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:17:00.754785 (+ 442us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:17:00.754788 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:17:00.755158 (+ 370us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:17:00.755312 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:17:00.755321 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:17:00.756342 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:17:00.756346 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:17:00.757249 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:17:00.757259 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:17:00.757972 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:17:00.757976 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:17:00.758092 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:17:00.758635 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:17:00.758650 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:17:00.762085 (+ 3435us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.217.62:44407: Server oryx/127.16.217.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":296,"threads_started":1}
W20260620 20:17:00.762493 26505 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:17:00.753651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.217.3:40055 (local address 127.16.217.62:44407)
0620 20:17:00.753856 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0620 20:17:00.753882 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0620 20:17:00.754301 (+ 419us) server_negotiation.cc:407] Connection header received
0620 20:17:00.754533 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:17:00.754539 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:17:00.754632 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:17:00.754748 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:17:00.755437 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:00.756229 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:00.757389 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:00.757785 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:00.757966 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:17:00.762262 (+ 4296us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.217.3:40055: BlockingRecv error: recv got EOF from 127.16.217.3:40055 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Jun 20 20:17:01 dist-test-slave-2hpg krb5kdc[25500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.217.1@KRBTEST.COM for oryx/127.16.217.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 20:17:01 dist-test-slave-2hpg krb5kdc[25500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.217.1@KRBTEST.COM for oryx/127.16.217.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 20:17:01.068609 26526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:01.059014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:44407 (local address 127.16.217.1:40513)
0620 20:17:01.059611 (+ 597us) negotiation.cc:107] Waiting for socket to connect
0620 20:17:01.059628 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 20:17:01.059766 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:17:01.060132 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:17:01.060135 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:17:01.060562 (+ 427us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:17:01.060779 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:17:01.060788 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:17:01.061947 (+ 1159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:17:01.061956 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:17:01.063081 (+ 1125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:17:01.063097 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:17:01.063831 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:17:01.063837 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:17:01.064019 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:17:01.064569 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:17:01.064585 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:17:01.068370 (+ 3785us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.217.62:44407: Server oryx/127.16.217.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":496,"thread_start_us":305,"threads_started":1}
W20260620 20:17:01.068845 26505 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:17:01.059115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.217.1:40513 (local address 127.16.217.62:44407)
0620 20:17:01.059385 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0620 20:17:01.059396 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 20:17:01.059728 (+ 332us) server_negotiation.cc:407] Connection header received
0620 20:17:01.059876 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:17:01.059880 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:17:01.059942 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:17:01.060024 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:17:01.060902 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:01.061793 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:01.063244 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:01.063684 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:01.063819 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:17:01.068581 (+ 4762us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.217.1:40513: BlockingRecv error: recv got EOF from 127.16.217.1:40513 (error 108)
Metrics: {"server-negotiator.queue_time_us":99}
Jun 20 20:17:01 dist-test-slave-2hpg krb5kdc[25500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.217.2@KRBTEST.COM for oryx/127.16.217.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 20:17:01 dist-test-slave-2hpg krb5kdc[25500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.217.2@KRBTEST.COM for oryx/127.16.217.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 20:17:01.449427 26527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:01.440376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:44407 (local address 127.16.217.2:42377)
0620 20:17:01.440983 (+ 607us) negotiation.cc:107] Waiting for socket to connect
0620 20:17:01.440999 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 20:17:01.441126 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:17:01.441468 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:17:01.441472 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:17:01.441906 (+ 434us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:17:01.442070 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:17:01.442079 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:17:01.443058 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:17:01.443061 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:17:01.444093 (+ 1032us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:17:01.444104 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:17:01.444879 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:17:01.444883 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:17:01.444991 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:17:01.445619 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:17:01.445635 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:17:01.449061 (+ 3426us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.217.62:44407: Server oryx/127.16.217.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":517,"thread_start_us":306,"threads_started":1}
W20260620 20:17:01.449563 26505 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:17:01.440405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.217.2:42377 (local address 127.16.217.62:44407)
0620 20:17:01.440636 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0620 20:17:01.440646 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 20:17:01.441086 (+ 440us) server_negotiation.cc:407] Connection header received
0620 20:17:01.441238 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:17:01.441242 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:17:01.441306 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:17:01.441395 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:17:01.442215 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:01.442933 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:01.444217 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:01.444732 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:01.444840 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:17:01.449283 (+ 4443us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.217.2:42377: BlockingRecv error: recv got EOF from 127.16.217.2:42377 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
Jun 20 20:17:01 dist-test-slave-2hpg krb5kdc[25500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.217.3@KRBTEST.COM for oryx/127.16.217.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 20:17:01 dist-test-slave-2hpg krb5kdc[25500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.217.3@KRBTEST.COM for oryx/127.16.217.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 20:17:01.772677 26528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:01.763342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:44407 (local address 127.16.217.3:42459)
0620 20:17:01.763968 (+ 626us) negotiation.cc:107] Waiting for socket to connect
0620 20:17:01.763984 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 20:17:01.764105 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:17:01.764424 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:17:01.764445 (+ 21us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:17:01.764854 (+ 409us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:17:01.765014 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:17:01.765022 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:17:01.766131 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:17:01.766135 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:17:01.767082 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:17:01.767093 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:17:01.767879 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:17:01.767884 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:17:01.767997 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:17:01.768645 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:17:01.768662 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:17:01.772302 (+ 3640us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.217.62:44407: Server oryx/127.16.217.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":535,"thread_start_us":304,"threads_started":1}
W20260620 20:17:01.772778 26505 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:17:01.763353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.217.3:42459 (local address 127.16.217.62:44407)
0620 20:17:01.763566 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0620 20:17:01.763575 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 20:17:01.764071 (+ 496us) server_negotiation.cc:407] Connection header received
0620 20:17:01.764210 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:17:01.764213 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:17:01.764292 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:17:01.764377 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:17:01.765180 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:01.765991 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:01.767236 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:01.767738 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:01.767846 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:17:01.772493 (+ 4647us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.217.3:42459: BlockingRecv error: recv got EOF from 127.16.217.3:42459 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Jun 20 20:17:02 dist-test-slave-2hpg krb5kdc[25500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.217.1@KRBTEST.COM for oryx/127.16.217.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 20:17:02 dist-test-slave-2hpg krb5kdc[25500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.217.1@KRBTEST.COM for oryx/127.16.217.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 20:17:02.079632 26529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:02.069679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:44407 (local address 127.16.217.1:36101)
0620 20:17:02.070290 (+ 611us) negotiation.cc:107] Waiting for socket to connect
0620 20:17:02.070308 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 20:17:02.070465 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:17:02.070848 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:17:02.070854 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:17:02.071286 (+ 432us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:17:02.071496 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:17:02.071509 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:17:02.072473 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:17:02.072477 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:17:02.073621 (+ 1144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:17:02.073634 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:17:02.074859 (+ 1225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:17:02.074866 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:17:02.075052 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:17:02.075748 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:17:02.075769 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:17:02.079274 (+ 3505us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.217.62:44407: Server oryx/127.16.217.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":333,"threads_started":1}
W20260620 20:17:02.079720 26505 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:17:02.069924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.217.1:36101 (local address 127.16.217.62:44407)
0620 20:17:02.070138 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0620 20:17:02.070145 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:17:02.070391 (+ 246us) server_negotiation.cc:407] Connection header received
0620 20:17:02.070602 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:17:02.070607 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:17:02.070685 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:17:02.070790 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:17:02.071641 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:02.072348 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:02.073795 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:02.074638 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:02.074786 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:17:02.079462 (+ 4676us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.217.1:36101: BlockingRecv error: recv got EOF from 127.16.217.1:36101 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
Jun 20 20:17:02 dist-test-slave-2hpg krb5kdc[25500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.217.2@KRBTEST.COM for oryx/127.16.217.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 20:17:02 dist-test-slave-2hpg krb5kdc[25500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.217.2@KRBTEST.COM for oryx/127.16.217.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 20:17:02.460566 26530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:02.450463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:44407 (local address 127.16.217.2:40103)
0620 20:17:02.451013 (+ 550us) negotiation.cc:107] Waiting for socket to connect
0620 20:17:02.451031 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 20:17:02.451205 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:17:02.451572 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:17:02.451577 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:17:02.452075 (+ 498us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:17:02.452334 (+ 259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:17:02.452346 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:17:02.453749 (+ 1403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:17:02.453756 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:17:02.454844 (+ 1088us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:17:02.454858 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:17:02.455649 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:17:02.455656 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:17:02.455851 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:17:02.456544 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:17:02.456564 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:17:02.460161 (+ 3597us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.217.62:44407: Server oryx/127.16.217.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":412,"thread_start_us":285,"threads_started":1}
W20260620 20:17:02.460682 26505 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:17:02.450547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.217.2:40103 (local address 127.16.217.62:44407)
0620 20:17:02.450842 (+ 295us) server_negotiation.cc:206] Beginning negotiation
0620 20:17:02.450849 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:17:02.451119 (+ 270us) server_negotiation.cc:407] Connection header received
0620 20:17:02.451340 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:17:02.451346 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:17:02.451426 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:17:02.451547 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:17:02.452495 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:02.453580 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:02.455000 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:02.455458 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:02.455613 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:17:02.460395 (+ 4782us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.217.2:40103: BlockingRecv error: recv got EOF from 127.16.217.2:40103 (error 108)
Metrics: {"server-negotiator.queue_time_us":138}
Jun 20 20:17:02 dist-test-slave-2hpg krb5kdc[25500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.217.3@KRBTEST.COM for oryx/127.16.217.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 20:17:02 dist-test-slave-2hpg krb5kdc[25500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.217.3@KRBTEST.COM for oryx/127.16.217.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 20:17:02.783437 26531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:02.773656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:44407 (local address 127.16.217.3:33861)
0620 20:17:02.774240 (+ 584us) negotiation.cc:107] Waiting for socket to connect
0620 20:17:02.774256 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 20:17:02.774378 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:17:02.774761 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:17:02.774766 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:17:02.775177 (+ 411us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:17:02.775396 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:17:02.775405 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:17:02.776495 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:17:02.776499 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:17:02.777431 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:17:02.777442 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:17:02.778348 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:17:02.778356 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:17:02.778555 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:17:02.779237 (+ 682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:17:02.779254 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:17:02.783173 (+ 3919us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.217.62:44407: Server oryx/127.16.217.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":415,"thread_start_us":310,"threads_started":1}
W20260620 20:17:02.783634 26505 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:17:02.773722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.217.3:33861 (local address 127.16.217.62:44407)
0620 20:17:02.773969 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0620 20:17:02.773978 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 20:17:02.774331 (+ 353us) server_negotiation.cc:407] Connection header received
0620 20:17:02.774472 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:17:02.774475 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:17:02.774539 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:17:02.774634 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:17:02.775523 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:02.776371 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:02.777586 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:02.778181 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:02.778276 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:17:02.783329 (+ 5053us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.217.3:33861: BlockingRecv error: recv got EOF from 127.16.217.3:33861 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Jun 20 20:17:03 dist-test-slave-2hpg krb5kdc[25500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.217.1@KRBTEST.COM for oryx/127.16.217.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 20:17:03 dist-test-slave-2hpg krb5kdc[25500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.217.1@KRBTEST.COM for oryx/127.16.217.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 20:17:03.089452 26532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:03.080620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:44407 (local address 127.16.217.1:39231)
0620 20:17:03.081098 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0620 20:17:03.081114 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 20:17:03.081267 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:17:03.081624 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:17:03.081628 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:17:03.082107 (+ 479us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:17:03.082250 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:17:03.082258 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:17:03.083231 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:17:03.083238 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:17:03.084289 (+ 1051us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:17:03.084301 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:17:03.084974 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:17:03.084979 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:17:03.085164 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:17:03.085758 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:17:03.085775 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:17:03.089099 (+ 3324us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.217.62:44407: Server oryx/127.16.217.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":255,"threads_started":1}
W20260620 20:17:03.089530 26505 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:17:03.080668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.217.1:39231 (local address 127.16.217.62:44407)
0620 20:17:03.080866 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0620 20:17:03.080873 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:17:03.081193 (+ 320us) server_negotiation.cc:407] Connection header received
0620 20:17:03.081429 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:17:03.081438 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:17:03.081498 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:17:03.081596 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:17:03.082384 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:03.083104 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:03.084420 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:03.084828 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:03.084959 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:17:03.089306 (+ 4347us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.217.1:39231: BlockingRecv error: recv got EOF from 127.16.217.1:39231 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Jun 20 20:17:03 dist-test-slave-2hpg krb5kdc[25500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.217.2@KRBTEST.COM for oryx/127.16.217.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 20:17:03 dist-test-slave-2hpg krb5kdc[25500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.217.2@KRBTEST.COM for oryx/127.16.217.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 20:17:03.470966 26533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:03.461670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:44407 (local address 127.16.217.2:50197)
0620 20:17:03.462178 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0620 20:17:03.462194 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 20:17:03.462358 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:17:03.462687 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:17:03.462691 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:17:03.463139 (+ 448us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:17:03.463279 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:17:03.463287 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:17:03.464433 (+ 1146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:17:03.464440 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:17:03.465556 (+ 1116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:17:03.465567 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:17:03.466330 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:17:03.466336 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:17:03.466501 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:17:03.467078 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:17:03.467094 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:17:03.470731 (+ 3637us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.217.62:44407: Server oryx/127.16.217.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":412,"thread_start_us":261,"threads_started":1}
W20260620 20:17:03.471194 26505 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:17:03.461757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.217.2:50197 (local address 127.16.217.62:44407)
0620 20:17:03.462019 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0620 20:17:03.462026 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:17:03.462279 (+ 253us) server_negotiation.cc:407] Connection header received
0620 20:17:03.462495 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:17:03.462499 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:17:03.462559 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:17:03.462656 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:17:03.463459 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:03.464322 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:03.465692 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:03.466170 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:03.466311 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:17:03.470943 (+ 4632us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.217.2:50197: BlockingRecv error: recv got EOF from 127.16.217.2:50197 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Jun 20 20:17:03 dist-test-slave-2hpg krb5kdc[25500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.217.3@KRBTEST.COM for oryx/127.16.217.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 20:17:03 dist-test-slave-2hpg krb5kdc[25500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.217.3@KRBTEST.COM for oryx/127.16.217.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 20:17:03.793875 26534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:03.784529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:44407 (local address 127.16.217.3:47173)
0620 20:17:03.785065 (+ 536us) negotiation.cc:107] Waiting for socket to connect
0620 20:17:03.785081 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 20:17:03.785202 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:17:03.785481 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:17:03.785484 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:17:03.786018 (+ 534us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:17:03.786178 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:17:03.786188 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:17:03.787388 (+ 1200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:17:03.787395 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:17:03.788331 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:17:03.788342 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:17:03.789067 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:17:03.789073 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:17:03.789260 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:17:03.789966 (+ 706us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:17:03.789984 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:17:03.793526 (+ 3542us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.217.62:44407: Server oryx/127.16.217.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":426,"thread_start_us":317,"threads_started":1}
W20260620 20:17:03.793987 26505 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:17:03.784724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.217.3:47173 (local address 127.16.217.62:44407)
0620 20:17:03.784899 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0620 20:17:03.784903 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:17:03.785147 (+ 244us) server_negotiation.cc:407] Connection header received
0620 20:17:03.785296 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:17:03.785299 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:17:03.785358 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:17:03.785440 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:17:03.786347 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:03.787241 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:03.788475 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:03.788907 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:03.789050 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:17:03.793733 (+ 4683us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.217.3:47173: BlockingRecv error: recv got EOF from 127.16.217.3:47173 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Jun 20 20:17:04 dist-test-slave-2hpg krb5kdc[25500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.217.1@KRBTEST.COM for oryx/127.16.217.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 20:17:04 dist-test-slave-2hpg krb5kdc[25500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.217.1@KRBTEST.COM for oryx/127.16.217.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 20:17:04.099135 26535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:04.090413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:44407 (local address 127.16.217.1:48817)
0620 20:17:04.090882 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0620 20:17:04.090899 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 20:17:04.091014 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:17:04.091347 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:17:04.091350 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:17:04.091705 (+ 355us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:17:04.091853 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:17:04.091861 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:17:04.092822 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:17:04.092826 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:17:04.093780 (+ 954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:17:04.093792 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:17:04.094528 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:17:04.094533 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:17:04.094645 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:17:04.095255 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:17:04.095273 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:17:04.098787 (+ 3514us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.217.62:44407: Server oryx/127.16.217.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":329,"thread_start_us":232,"threads_started":1}
W20260620 20:17:04.099244 26505 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:17:04.090509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.217.1:48817 (local address 127.16.217.62:44407)
0620 20:17:04.090773 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0620 20:17:04.090779 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:17:04.090972 (+ 193us) server_negotiation.cc:407] Connection header received
0620 20:17:04.091128 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:17:04.091134 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:17:04.091217 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:17:04.091294 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:17:04.091977 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:04.092692 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:04.093948 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:04.094385 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:04.094484 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:17:04.098991 (+ 4507us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.217.1:48817: BlockingRecv error: recv got EOF from 127.16.217.1:48817 (error 108)
Metrics: {"server-negotiator.queue_time_us":130}
Jun 20 20:17:04 dist-test-slave-2hpg krb5kdc[25500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.217.2@KRBTEST.COM for oryx/127.16.217.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 20:17:04 dist-test-slave-2hpg krb5kdc[25500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.217.2@KRBTEST.COM for oryx/127.16.217.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 20:17:04.485526 26536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:04.471867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:44407 (local address 127.16.217.2:50045)
0620 20:17:04.472390 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0620 20:17:04.472406 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 20:17:04.472516 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:17:04.472801 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:17:04.472804 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:17:04.473142 (+ 338us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:17:04.473295 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:17:04.473304 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:17:04.474462 (+ 1158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:17:04.474467 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:17:04.475613 (+ 1146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:17:04.475640 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:17:04.476359 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:17:04.476364 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:17:04.476484 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:17:04.481549 (+ 5065us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:17:04.481577 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:17:04.485153 (+ 3576us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.217.62:44407: Server oryx/127.16.217.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":295,"threads_started":1}
W20260620 20:17:04.485605 26505 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:17:04.471956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.217.2:50045 (local address 127.16.217.62:44407)
0620 20:17:04.472136 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0620 20:17:04.472143 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:17:04.472486 (+ 343us) server_negotiation.cc:407] Connection header received
0620 20:17:04.472611 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:17:04.472615 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:17:04.472675 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:17:04.472759 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:17:04.473410 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:04.474301 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:04.475784 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:04.476193 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:04.476301 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:17:04.485346 (+ 9045us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.217.2:50045: BlockingRecv error: recv got EOF from 127.16.217.2:50045 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Jun 20 20:17:04 dist-test-slave-2hpg krb5kdc[25500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.217.3@KRBTEST.COM for oryx/127.16.217.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 20:17:04 dist-test-slave-2hpg krb5kdc[25500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.217.3@KRBTEST.COM for oryx/127.16.217.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 20:17:04.803678 26537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:04.794758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:44407 (local address 127.16.217.3:46827)
0620 20:17:04.795214 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0620 20:17:04.795230 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 20:17:04.795339 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:17:04.795622 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:17:04.795625 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:17:04.796082 (+ 457us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:17:04.796248 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:17:04.796257 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:17:04.797351 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:17:04.797355 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:17:04.798343 (+ 988us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:17:04.798354 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:17:04.799087 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:17:04.799092 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:17:04.799245 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:17:04.799871 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:17:04.799888 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:17:04.803422 (+ 3534us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.217.62:44407: Server oryx/127.16.217.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":353,"thread_start_us":225,"threads_started":1}
W20260620 20:17:04.803915 26505 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:17:04.794957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.217.3:46827 (local address 127.16.217.62:44407)
0620 20:17:04.795239 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0620 20:17:04.795244 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:17:04.795282 (+ 38us) server_negotiation.cc:407] Connection header received
0620 20:17:04.795438 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:17:04.795442 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:17:04.795501 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:17:04.795583 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:17:04.796406 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:04.797210 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:04.798477 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:04.798898 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:04.798995 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:17:04.803659 (+ 4664us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.217.3:46827: BlockingRecv error: recv got EOF from 127.16.217.3:46827 (error 108)
Metrics: {"server-negotiator.queue_time_us":132}
Jun 20 20:17:05 dist-test-slave-2hpg krb5kdc[25500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.217.1@KRBTEST.COM for oryx/127.16.217.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 20:17:05 dist-test-slave-2hpg krb5kdc[25500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.217.1@KRBTEST.COM for oryx/127.16.217.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 20:17:05.109129 26538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:05.100185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:44407 (local address 127.16.217.1:42641)
0620 20:17:05.100711 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0620 20:17:05.100727 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 20:17:05.100894 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:17:05.101255 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:17:05.101261 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:17:05.101765 (+ 504us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:17:05.101966 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:17:05.101975 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:17:05.102937 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:17:05.102945 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:17:05.104020 (+ 1075us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:17:05.104031 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:17:05.104805 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:17:05.104810 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:17:05.104983 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:17:05.105527 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:17:05.105543 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:17:05.108814 (+ 3271us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.217.62:44407: Server oryx/127.16.217.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":388,"thread_start_us":265,"threads_started":1}
W20260620 20:17:05.109242 26505 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:17:05.100292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.217.1:42641 (local address 127.16.217.62:44407)
0620 20:17:05.100540 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0620 20:17:05.100548 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 20:17:05.100817 (+ 269us) server_negotiation.cc:407] Connection header received
0620 20:17:05.101021 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:17:05.101032 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:17:05.101103 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:17:05.101210 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:17:05.102109 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:05.102812 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:05.104150 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:05.104654 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:05.104790 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:17:05.109015 (+ 4225us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.217.1:42641: BlockingRecv error: recv got EOF from 127.16.217.1:42641 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Jun 20 20:17:05 dist-test-slave-2hpg krb5kdc[25500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.217.2@KRBTEST.COM for oryx/127.16.217.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 20:17:05 dist-test-slave-2hpg krb5kdc[25500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.217.2@KRBTEST.COM for oryx/127.16.217.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 20:17:05.496804 26539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:05.486600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:44407 (local address 127.16.217.2:46685)
0620 20:17:05.487325 (+ 725us) negotiation.cc:107] Waiting for socket to connect
0620 20:17:05.487341 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 20:17:05.487452 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:17:05.487803 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:17:05.487807 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:17:05.488266 (+ 459us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:17:05.488448 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:17:05.488457 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:17:05.489813 (+ 1356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:17:05.489818 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:17:05.490887 (+ 1069us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:17:05.490902 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:17:05.491686 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:17:05.491692 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:17:05.491894 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:17:05.492534 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:17:05.492554 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:17:05.496372 (+ 3818us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.217.62:44407: Server oryx/127.16.217.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":610,"thread_start_us":514,"threads_started":1}
W20260620 20:17:05.496832 26505 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:17:05.486802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.217.2:46685 (local address 127.16.217.62:44407)
0620 20:17:05.487038 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0620 20:17:05.487048 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 20:17:05.487420 (+ 372us) server_negotiation.cc:407] Connection header received
0620 20:17:05.487594 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:17:05.487598 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:17:05.487659 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:17:05.487751 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:17:05.488581 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:05.489622 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:05.491050 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:05.491511 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:05.491678 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:17:05.496557 (+ 4879us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.217.2:46685: BlockingRecv error: recv got EOF from 127.16.217.2:46685 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Jun 20 20:17:05 dist-test-slave-2hpg krb5kdc[25500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.217.3@KRBTEST.COM for oryx/127.16.217.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 20:17:05 dist-test-slave-2hpg krb5kdc[25500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.217.3@KRBTEST.COM for oryx/127.16.217.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 20:17:05.814023 26540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:05.804660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:44407 (local address 127.16.217.3:35969)
0620 20:17:05.805167 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0620 20:17:05.805187 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 20:17:05.805353 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:17:05.805689 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:17:05.805693 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:17:05.806206 (+ 513us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:17:05.806383 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:17:05.806392 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:17:05.807391 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:17:05.807396 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:17:05.808323 (+ 927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:17:05.808333 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:17:05.809015 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:17:05.809021 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:17:05.809200 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:17:05.809798 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:17:05.809815 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:17:05.813726 (+ 3911us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.217.62:44407: Server oryx/127.16.217.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":393,"thread_start_us":230,"threads_started":1}
W20260620 20:17:05.814177 26505 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:17:05.804730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.217.3:35969 (local address 127.16.217.62:44407)
0620 20:17:05.804937 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0620 20:17:05.804945 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 20:17:05.805273 (+ 328us) server_negotiation.cc:407] Connection header received
0620 20:17:05.805474 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:17:05.805480 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:17:05.805551 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:17:05.805699 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:17:05.806524 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:05.807255 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:05.808473 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:05.808874 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:05.809004 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:17:05.813949 (+ 4945us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.217.3:35969: BlockingRecv error: recv got EOF from 127.16.217.3:35969 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Jun 20 20:17:06 dist-test-slave-2hpg krb5kdc[25500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.217.1@KRBTEST.COM for oryx/127.16.217.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 20:17:06 dist-test-slave-2hpg krb5kdc[25500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.217.1@KRBTEST.COM for oryx/127.16.217.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 20:17:06.120414 26541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:06.110235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:44407 (local address 127.16.217.1:54397)
0620 20:17:06.110831 (+ 596us) negotiation.cc:107] Waiting for socket to connect
0620 20:17:06.110850 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 20:17:06.110974 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:17:06.111306 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:17:06.111309 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:17:06.111762 (+ 453us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:17:06.111943 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:17:06.111951 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:17:06.113375 (+ 1424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:17:06.113382 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:17:06.114842 (+ 1460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:17:06.114855 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:17:06.115581 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:17:06.115585 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:17:06.115750 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:17:06.116470 (+ 720us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:17:06.116489 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:17:06.120044 (+ 3555us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.217.62:44407: Server oryx/127.16.217.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":431,"thread_start_us":327,"threads_started":1}
W20260620 20:17:06.120476 26505 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:17:06.110342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.217.1:54397 (local address 127.16.217.62:44407)
0620 20:17:06.110534 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0620 20:17:06.110538 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:17:06.110962 (+ 424us) server_negotiation.cc:407] Connection header received
0620 20:17:06.111083 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:17:06.111087 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:17:06.111152 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:17:06.111248 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:17:06.112113 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:06.113200 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:06.114981 (+ 1781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:06.115453 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:06.115578 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:17:06.120254 (+ 4676us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.217.1:54397: BlockingRecv error: recv got EOF from 127.16.217.1:54397 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Jun 20 20:17:06 dist-test-slave-2hpg krb5kdc[25500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.217.2@KRBTEST.COM for oryx/127.16.217.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 20:17:06 dist-test-slave-2hpg krb5kdc[25500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.217.2@KRBTEST.COM for oryx/127.16.217.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 20:17:06.506860 26542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:06.497863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:44407 (local address 127.16.217.2:54511)
0620 20:17:06.498452 (+ 589us) negotiation.cc:107] Waiting for socket to connect
0620 20:17:06.498468 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 20:17:06.498579 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:17:06.498972 (+ 393us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:17:06.498975 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:17:06.499387 (+ 412us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:17:06.499544 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:17:06.499552 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:17:06.500554 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:17:06.500558 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:17:06.501536 (+ 978us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:17:06.501547 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:17:06.502295 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:17:06.502299 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:17:06.502429 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:17:06.502982 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:17:06.502997 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:17:06.506573 (+ 3576us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.217.62:44407: Server oryx/127.16.217.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":325,"threads_started":1}
W20260620 20:17:06.507063 26505 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:17:06.497913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.217.2:54511 (local address 127.16.217.62:44407)
0620 20:17:06.498092 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0620 20:17:06.498097 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:17:06.498541 (+ 444us) server_negotiation.cc:407] Connection header received
0620 20:17:06.498730 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:17:06.498735 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:17:06.498808 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:17:06.498910 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:17:06.499683 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:06.500435 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:06.501655 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:06.502157 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:06.502258 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:17:06.506826 (+ 4568us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.217.2:54511: BlockingRecv error: recv got EOF from 127.16.217.2:54511 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Jun 20 20:17:06 dist-test-slave-2hpg krb5kdc[25500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.217.3@KRBTEST.COM for oryx/127.16.217.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 20:17:06 dist-test-slave-2hpg krb5kdc[25500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.217.3@KRBTEST.COM for oryx/127.16.217.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 20:17:06.824494 26543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:06.815065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:44407 (local address 127.16.217.3:43383)
0620 20:17:06.815581 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0620 20:17:06.815600 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 20:17:06.815769 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:17:06.816086 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:17:06.816090 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:17:06.816458 (+ 368us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:17:06.816621 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:17:06.816629 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:17:06.817694 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:17:06.817697 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:17:06.818674 (+ 977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:17:06.818690 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:17:06.819766 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:17:06.819772 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:17:06.819965 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:17:06.820636 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:17:06.820656 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:17:06.824156 (+ 3500us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.217.62:44407: Server oryx/127.16.217.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":297,"threads_started":1}
W20260620 20:17:06.824674 26505 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:17:06.815143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.217.3:43383 (local address 127.16.217.62:44407)
0620 20:17:06.815429 (+ 286us) server_negotiation.cc:206] Beginning negotiation
0620 20:17:06.815436 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:17:06.815747 (+ 311us) server_negotiation.cc:407] Connection header received
0620 20:17:06.815877 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:17:06.815881 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:17:06.815941 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:17:06.816024 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:17:06.816790 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:06.817543 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:06.818876 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:06.819561 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:06.819725 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:17:06.824416 (+ 4691us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.217.3:43383: BlockingRecv error: recv got EOF from 127.16.217.3:43383 (error 108)
Metrics: {"server-negotiator.queue_time_us":117}
Jun 20 20:17:07 dist-test-slave-2hpg krb5kdc[25500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.217.1@KRBTEST.COM for oryx/127.16.217.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 20:17:07 dist-test-slave-2hpg krb5kdc[25500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.217.1@KRBTEST.COM for oryx/127.16.217.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 20:17:07.130309 26544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:07.121370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:44407 (local address 127.16.217.1:49565)
0620 20:17:07.121948 (+ 578us) negotiation.cc:107] Waiting for socket to connect
0620 20:17:07.121970 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 20:17:07.122087 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:17:07.122427 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:17:07.122431 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:17:07.122848 (+ 417us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:17:07.122990 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:17:07.122998 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:17:07.124044 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:17:07.124048 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:17:07.124955 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:17:07.124966 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:17:07.125655 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:17:07.125661 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:17:07.125793 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:17:07.126494 (+ 701us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:17:07.126511 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:17:07.129973 (+ 3462us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.217.62:44407: Server oryx/127.16.217.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":433,"thread_start_us":338,"threads_started":1}
W20260620 20:17:07.130352 26505 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:17:07.121524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.217.1:49565 (local address 127.16.217.62:44407)
0620 20:17:07.121789 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0620 20:17:07.121794 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:17:07.122047 (+ 253us) server_negotiation.cc:407] Connection header received
0620 20:17:07.122188 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:17:07.122192 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:17:07.122256 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:17:07.122347 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:17:07.123139 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:07.123910 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:07.125103 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:07.125518 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:07.125620 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:17:07.130167 (+ 4547us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.217.1:49565: BlockingRecv error: recv got EOF from 127.16.217.1:49565 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Jun 20 20:17:07 dist-test-slave-2hpg krb5kdc[25500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.217.2@KRBTEST.COM for oryx/127.16.217.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 20:17:07 dist-test-slave-2hpg krb5kdc[25500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.217.2@KRBTEST.COM for oryx/127.16.217.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 20:17:07.516960 26545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:07.507718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:44407 (local address 127.16.217.2:47915)
0620 20:17:07.508241 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0620 20:17:07.508256 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 20:17:07.508409 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:17:07.508781 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:17:07.508785 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:17:07.509230 (+ 445us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:17:07.509406 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:17:07.509415 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:17:07.510642 (+ 1227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:17:07.510649 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:17:07.511577 (+ 928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:17:07.511587 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:17:07.512300 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:17:07.512306 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:17:07.512493 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:17:07.513077 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:17:07.513094 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:17:07.516605 (+ 3511us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.217.62:44407: Server oryx/127.16.217.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":417,"thread_start_us":308,"threads_started":1}
W20260620 20:17:07.517040 26505 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:17:07.507912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.217.2:47915 (local address 127.16.217.62:44407)
0620 20:17:07.508133 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0620 20:17:07.508139 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:17:07.508336 (+ 197us) server_negotiation.cc:407] Connection header received
0620 20:17:07.508550 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:17:07.508554 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:17:07.508613 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:17:07.508712 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:17:07.509541 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:07.510500 (+ 959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:07.511717 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:07.512127 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:07.512277 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:17:07.516807 (+ 4530us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.217.2:47915: BlockingRecv error: recv got EOF from 127.16.217.2:47915 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Jun 20 20:17:07 dist-test-slave-2hpg krb5kdc[25500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.217.3@KRBTEST.COM for oryx/127.16.217.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 20:17:07 dist-test-slave-2hpg krb5kdc[25500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.217.3@KRBTEST.COM for oryx/127.16.217.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 20:17:07.834369 26546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:07.825399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:44407 (local address 127.16.217.3:60537)
0620 20:17:07.825975 (+ 576us) negotiation.cc:107] Waiting for socket to connect
0620 20:17:07.825992 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 20:17:07.826104 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:17:07.826418 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:17:07.826421 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:17:07.826822 (+ 401us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:17:07.826965 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:17:07.826973 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:17:07.827906 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:17:07.827911 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:17:07.828888 (+ 977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:17:07.828899 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:17:07.829566 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:17:07.829571 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:17:07.829730 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:17:07.830603 (+ 873us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:17:07.830629 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:17:07.834109 (+ 3480us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.217.62:44407: Server oryx/127.16.217.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":336,"threads_started":1}
W20260620 20:17:07.834462 26505 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:17:07.825532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.217.3:60537 (local address 127.16.217.62:44407)
0620 20:17:07.825695 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0620 20:17:07.825725 (+ 30us) server_negotiation.cc:399] Waiting for connection header
0620 20:17:07.826065 (+ 340us) server_negotiation.cc:407] Connection header received
0620 20:17:07.826198 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:17:07.826203 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:17:07.826274 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:17:07.826360 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:17:07.827084 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:07.827777 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:07.829020 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:07.829434 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:07.829526 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:17:07.834294 (+ 4768us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.217.3:60537: BlockingRecv error: recv got EOF from 127.16.217.3:60537 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Jun 20 20:17:08 dist-test-slave-2hpg krb5kdc[25500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.217.1@KRBTEST.COM for oryx/127.16.217.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 20:17:08 dist-test-slave-2hpg krb5kdc[25500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.217.1@KRBTEST.COM for oryx/127.16.217.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 20:17:08.140483 26547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:08.131548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:44407 (local address 127.16.217.1:49389)
0620 20:17:08.132074 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0620 20:17:08.132089 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 20:17:08.132241 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:17:08.132563 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:17:08.132567 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:17:08.132980 (+ 413us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:17:08.133136 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:17:08.133144 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:17:08.134217 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:17:08.134224 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:17:08.135172 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:17:08.135182 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:17:08.135876 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:17:08.135882 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:17:08.136060 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:17:08.136673 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:17:08.136688 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:17:08.140127 (+ 3439us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.217.62:44407: Server oryx/127.16.217.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":411,"thread_start_us":293,"threads_started":1}
W20260620 20:17:08.140610 26505 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:17:08.131631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.217.1:49389 (local address 127.16.217.62:44407)
0620 20:17:08.131952 (+ 321us) server_negotiation.cc:206] Beginning negotiation
0620 20:17:08.131958 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:17:08.132171 (+ 213us) server_negotiation.cc:407] Connection header received
0620 20:17:08.132368 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:17:08.132377 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:17:08.132437 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:17:08.132535 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:17:08.133283 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:08.134079 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:08.135311 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:08.135736 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:08.135867 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:17:08.140325 (+ 4458us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.217.1:49389: BlockingRecv error: recv got EOF from 127.16.217.1:49389 (error 108)
Metrics: {"server-negotiator.queue_time_us":145}
Jun 20 20:17:08 dist-test-slave-2hpg krb5kdc[25500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.217.2@KRBTEST.COM for oryx/127.16.217.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 20:17:08 dist-test-slave-2hpg krb5kdc[25500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.217.2@KRBTEST.COM for oryx/127.16.217.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 20:17:08.527611 26548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:08.518300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:44407 (local address 127.16.217.2:56717)
0620 20:17:08.518819 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0620 20:17:08.518835 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 20:17:08.518987 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:17:08.519323 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:17:08.519328 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:17:08.519733 (+ 405us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:17:08.519897 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:17:08.519906 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:17:08.520907 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:17:08.520914 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:17:08.521908 (+ 994us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:17:08.521918 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:17:08.522593 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:17:08.522599 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:17:08.522780 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:17:08.523397 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:17:08.523414 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:17:08.527253 (+ 3839us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.217.62:44407: Server oryx/127.16.217.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":398,"thread_start_us":241,"threads_started":1}
W20260620 20:17:08.527704 26505 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:17:08.518429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.217.2:56717 (local address 127.16.217.62:44407)
0620 20:17:08.518633 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0620 20:17:08.518639 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:17:08.518923 (+ 284us) server_negotiation.cc:407] Connection header received
0620 20:17:08.519108 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:17:08.519114 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:17:08.519198 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:17:08.519299 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:17:08.520039 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:08.520768 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:08.522046 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:08.522446 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:08.522584 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:17:08.527469 (+ 4885us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.217.2:56717: BlockingRecv error: recv got EOF from 127.16.217.2:56717 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
I20260620 20:17:08.538586 17252 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu pid 26087
I20260620 20:17:08.551323 26550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:08.539687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45392 (local address 127.16.217.1:34899)
0620 20:17:08.540355 (+ 668us) server_negotiation.cc:206] Beginning negotiation
0620 20:17:08.540362 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:17:08.540388 (+ 26us) server_negotiation.cc:407] Connection header received
0620 20:17:08.540450 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:17:08.540456 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:17:08.540747 (+ 291us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:17:08.540917 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:17:08.542385 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:08.543531 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:08.544838 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:08.545518 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:08.545638 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:17:08.548156 (+ 2518us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:17:08.548165 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:17:08.548179 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:17:08.548220 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:17:08.550831 (+ 2611us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.16.217.1@KRBTEST.COM not found in keytab (ticket kvno 2)
0620 20:17:08.551099 (+ 268us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:45392: Request ticket server kudu/127.16.217.1@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":490,"thread_start_us":339,"threads_started":1}
W20260620 20:17:08.551450 26550 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:45392: Request ticket server kudu/127.16.217.1@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260620 20:17:08.551456 17252 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu pid 26087: Not authorized: Client connection negotiation failed: client connection to 127.16.217.1:34899: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.16.217.1@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260620 20:17:08.551641 17252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu with pid 26087
I20260620 20:17:08.568987 17252 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu pid 26238
I20260620 20:17:08.581599 26551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:08.570048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57288 (local address 127.16.217.2:33487)
0620 20:17:08.570670 (+ 622us) server_negotiation.cc:206] Beginning negotiation
0620 20:17:08.570680 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 20:17:08.570716 (+ 36us) server_negotiation.cc:407] Connection header received
0620 20:17:08.570788 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:17:08.570797 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:17:08.571099 (+ 302us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:17:08.571326 (+ 227us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:17:08.572502 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:08.573758 (+ 1256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:08.575169 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:08.575876 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:08.575974 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:17:08.578361 (+ 2387us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:17:08.578369 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:17:08.578383 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:17:08.578422 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:17:08.581033 (+ 2611us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.16.217.2@KRBTEST.COM not found in keytab (ticket kvno 2)
0620 20:17:08.581306 (+ 273us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:57288: Request ticket server kudu/127.16.217.2@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":419,"thread_start_us":271,"threads_started":1}
W20260620 20:17:08.581697 26551 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:57288: Request ticket server kudu/127.16.217.2@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260620 20:17:08.581983 17252 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu pid 26238: Not authorized: Client connection negotiation failed: client connection to 127.16.217.2:33487: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.16.217.2@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260620 20:17:08.582135 17252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu with pid 26238
I20260620 20:17:08.598769 17252 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu pid 26371
I20260620 20:17:08.611263 26552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:08.599797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37522 (local address 127.16.217.3:36363)
0620 20:17:08.600399 (+ 602us) server_negotiation.cc:206] Beginning negotiation
0620 20:17:08.600406 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:17:08.600431 (+ 25us) server_negotiation.cc:407] Connection header received
0620 20:17:08.600494 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:17:08.600500 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:17:08.600751 (+ 251us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:17:08.600903 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:17:08.602954 (+ 2051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:08.603944 (+ 990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:08.605143 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:08.605697 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:08.605844 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:17:08.608233 (+ 2389us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:17:08.608240 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:17:08.608254 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:17:08.608288 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:17:08.610740 (+ 2452us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.16.217.3@KRBTEST.COM not found in keytab (ticket kvno 2)
0620 20:17:08.610993 (+ 253us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:37522: Request ticket server kudu/127.16.217.3@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":448,"thread_start_us":280,"threads_started":1}
W20260620 20:17:08.611366 26552 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:37522: Request ticket server kudu/127.16.217.3@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260620 20:17:08.611413 17252 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu pid 26371: Not authorized: Client connection negotiation failed: client connection to 127.16.217.3:36363: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.16.217.3@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260620 20:17:08.611588 17252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu with pid 26371
I20260620 20:17:08.628031 17252 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu pid 26018
I20260620 20:17:08.629428 26080 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:17:08.751194 17252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu with pid 26018
2026-06-20T20:17:08Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (21191 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-06-20T20:17:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T20:17:08Z Disabled control of system clock
I20260620 20:17:08.827881 17252 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
/tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986536687152-17252-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986536687152-17252-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986536687152-17252-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986536687152-17252-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.217.62:34795
--webserver_interface=127.16.217.62
--webserver_port=0
--builtin_ntp_servers=127.16.217.20:34201
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.217.62:34795
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260620 20:17:09.156042 26557 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:17:09.156332 26557 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:17:09.156409 26557 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:17:09.164522 26557 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:17:09.164686 26557 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:17:09.164757 26557 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:17:09.164839 26557 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:17:09.164903 26557 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:17:09.176139 26557 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.217.20:34201
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986536687152-17252-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986536687152-17252-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.217.62:34795
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.217.62:34795
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986536687152-17252-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.217.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986536687152-17252-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:17:09.177474 26557 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:17:09.179293 26557 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:17:09.191457 26562 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:17:09.191454 26563 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:17:09.192257 26557 server_base.cc:1034] running on GCE node
W20260620 20:17:09.192909 26565 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:17:09.193991 26557 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:17:09.195889 26557 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:17:09.197104 26557 hybrid_clock.cc:648] HybridClock initialized: now 1781986629197059 us; error 33 us; skew 500 ppm
I20260620 20:17:09.197592 26557 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 20:17:09.200588 26557 webserver.cc:460] Webserver started at http://127.16.217.62:33867/ using document root <none> and password file <none>
I20260620 20:17:09.201332 26557 fs_manager.cc:359] Metadata directory not provided
I20260620 20:17:09.201463 26557 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:17:09.201787 26557 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:17:09.204625 26557 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986536687152-17252-0/minicluster-data/master-0/data/instance:
uuid: "f6abfe47ca1248a68bd1f1d5aa379e65"
format_stamp: "Formatted at 2026-06-20 20:17:09 on dist-test-slave-2hpg"
server_key: "18f1f323603ccc72729c6aefb9e510ce"
server_key_iv: "670d6af7d5f47b3068f786ca25a647d8"
server_key_version: "encryptionkey@0"
I20260620 20:17:09.205421 26557 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986536687152-17252-0/minicluster-data/master-0/wal/instance:
uuid: "f6abfe47ca1248a68bd1f1d5aa379e65"
format_stamp: "Formatted at 2026-06-20 20:17:09 on dist-test-slave-2hpg"
server_key: "18f1f323603ccc72729c6aefb9e510ce"
server_key_iv: "670d6af7d5f47b3068f786ca25a647d8"
server_key_version: "encryptionkey@0"
I20260620 20:17:09.210418 26557 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260620 20:17:09.213697 26571 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:17:09.215046 26557 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 20:17:09.215221 26557 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986536687152-17252-0/minicluster-data/master-0/data,/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986536687152-17252-0/minicluster-data/master-0/wal
uuid: "f6abfe47ca1248a68bd1f1d5aa379e65"
format_stamp: "Formatted at 2026-06-20 20:17:09 on dist-test-slave-2hpg"
server_key: "18f1f323603ccc72729c6aefb9e510ce"
server_key_iv: "670d6af7d5f47b3068f786ca25a647d8"
server_key_version: "encryptionkey@0"
I20260620 20:17:09.215432 26557 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986536687152-17252-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986536687152-17252-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986536687152-17252-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:17:09.226647 26557 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:17:09.227424 26557 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:17:09.227718 26557 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:17:09.245703 26557 rpc_server.cc:307] RPC server started. Bound to: 127.16.217.62:34795
I20260620 20:17:09.245731 26622 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.217.62:34795 every 8 connection(s)
I20260620 20:17:09.247028 26557 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986536687152-17252-0/minicluster-data/master-0/data/info.pb
I20260620 20:17:09.250583 26623 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:17:09.251444 17252 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu as pid 26557
I20260620 20:17:09.251626 17252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986536687152-17252-0/minicluster-data/master-0/wal/instance
I20260620 20:17:09.252282 17252 external_mini_cluster.cc:1428] Setting key 32dbd9094a16e65858b640c593cf3ae4
I20260620 20:17:09.263543 26623 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f6abfe47ca1248a68bd1f1d5aa379e65: Bootstrap starting.
I20260620 20:17:09.265741 26626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:09.254094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56928 (local address 127.16.217.62:34795)
0620 20:17:09.254928 (+ 834us) server_negotiation.cc:206] Beginning negotiation
0620 20:17:09.254938 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 20:17:09.254977 (+ 39us) server_negotiation.cc:407] Connection header received
0620 20:17:09.257185 (+ 2208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:17:09.257214 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:17:09.257684 (+ 470us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:17:09.258138 (+ 454us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:17:09.258693 (+ 555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:09.259920 (+ 1227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:09.261266 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:09.262054 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:09.262257 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:17:09.263463 (+ 1206us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:17:09.263470 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:17:09.263484 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:17:09.264078 (+ 594us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:17:09.264155 (+ 77us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:17:09.264175 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:17:09.264306 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0620 20:17:09.264489 (+ 183us) server_negotiation.cc:299] Negotiation successful
0620 20:17:09.264795 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":434,"thread_start_us":262,"threads_started":1}
I20260620 20:17:09.266875 26623 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f6abfe47ca1248a68bd1f1d5aa379e65: Neither blocks nor log segments found. Creating new log.
I20260620 20:17:09.268255 26623 log.cc:824] T 00000000000000000000000000000000 P f6abfe47ca1248a68bd1f1d5aa379e65: Log is configured to *not* fsync() on all Append() calls
I20260620 20:17:09.271209 26623 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f6abfe47ca1248a68bd1f1d5aa379e65: No bootstrap required, opened a new log
I20260620 20:17:09.278916 26623 raft_consensus.cc:348] T 00000000000000000000000000000000 P f6abfe47ca1248a68bd1f1d5aa379e65 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6abfe47ca1248a68bd1f1d5aa379e65" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 34795 } }
I20260620 20:17:09.279273 26623 raft_consensus.cc:374] T 00000000000000000000000000000000 P f6abfe47ca1248a68bd1f1d5aa379e65 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:17:09.279343 26623 raft_consensus.cc:729] T 00000000000000000000000000000000 P f6abfe47ca1248a68bd1f1d5aa379e65 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6abfe47ca1248a68bd1f1d5aa379e65, State: Initialized, Role: FOLLOWER
I20260620 20:17:09.279949 26623 consensus_queue.cc:260] T 00000000000000000000000000000000 P f6abfe47ca1248a68bd1f1d5aa379e65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6abfe47ca1248a68bd1f1d5aa379e65" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 34795 } }
I20260620 20:17:09.280248 26623 raft_consensus.cc:388] T 00000000000000000000000000000000 P f6abfe47ca1248a68bd1f1d5aa379e65 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 20:17:09.280418 26623 raft_consensus.cc:482] T 00000000000000000000000000000000 P f6abfe47ca1248a68bd1f1d5aa379e65 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 20:17:09.280597 26623 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f6abfe47ca1248a68bd1f1d5aa379e65 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:17:09.282927 26623 raft_consensus.cc:504] T 00000000000000000000000000000000 P f6abfe47ca1248a68bd1f1d5aa379e65 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6abfe47ca1248a68bd1f1d5aa379e65" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 34795 } }
I20260620 20:17:09.283429 26623 leader_election.cc:302] T 00000000000000000000000000000000 P f6abfe47ca1248a68bd1f1d5aa379e65 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f6abfe47ca1248a68bd1f1d5aa379e65; no voters:
I20260620 20:17:09.284018 26623 leader_election.cc:288] T 00000000000000000000000000000000 P f6abfe47ca1248a68bd1f1d5aa379e65 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 20:17:09.284158 26628 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f6abfe47ca1248a68bd1f1d5aa379e65 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:17:09.284848 26628 raft_consensus.cc:686] T 00000000000000000000000000000000 P f6abfe47ca1248a68bd1f1d5aa379e65 [term 1 LEADER]: Becoming Leader. State: Replica: f6abfe47ca1248a68bd1f1d5aa379e65, State: Running, Role: LEADER
I20260620 20:17:09.285354 26628 consensus_queue.cc:237] T 00000000000000000000000000000000 P f6abfe47ca1248a68bd1f1d5aa379e65 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6abfe47ca1248a68bd1f1d5aa379e65" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 34795 } }
I20260620 20:17:09.286165 26623 sys_catalog.cc:564] T 00000000000000000000000000000000 P f6abfe47ca1248a68bd1f1d5aa379e65 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:17:09.288507 26630 sys_catalog.cc:455] T 00000000000000000000000000000000 P f6abfe47ca1248a68bd1f1d5aa379e65 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f6abfe47ca1248a68bd1f1d5aa379e65. Latest consensus state: current_term: 1 leader_uuid: "f6abfe47ca1248a68bd1f1d5aa379e65" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6abfe47ca1248a68bd1f1d5aa379e65" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 34795 } } }
I20260620 20:17:09.288892 26630 sys_catalog.cc:458] T 00000000000000000000000000000000 P f6abfe47ca1248a68bd1f1d5aa379e65 [sys.catalog]: This master's current role is: LEADER
I20260620 20:17:09.289433 26629 sys_catalog.cc:455] T 00000000000000000000000000000000 P f6abfe47ca1248a68bd1f1d5aa379e65 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f6abfe47ca1248a68bd1f1d5aa379e65" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6abfe47ca1248a68bd1f1d5aa379e65" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 34795 } } }
I20260620 20:17:09.289750 26629 sys_catalog.cc:458] T 00000000000000000000000000000000 P f6abfe47ca1248a68bd1f1d5aa379e65 [sys.catalog]: This master's current role is: LEADER
I20260620 20:17:09.289825 26633 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 20:17:09.295182 26633 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 20:17:09.303305 26633 catalog_manager.cc:1347] Generated new cluster ID: d802261c446a41189403e13bfe63f986
I20260620 20:17:09.303416 26633 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 20:17:09.321228 26633 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 20:17:09.323225 26633 catalog_manager.cc:1504] Loading token signing keys...
I20260620 20:17:09.337574 26633 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f6abfe47ca1248a68bd1f1d5aa379e65: Generated new TSK 0
I20260620 20:17:09.338531 26633 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 20:17:09.353762 17252 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
/tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986536687152-17252-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986536687152-17252-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986536687152-17252-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986536687152-17252-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.217.1:0
--local_ip_for_outbound_sockets=127.16.217.1
--webserver_interface=127.16.217.1
--webserver_port=0
--tserver_master_addrs=127.16.217.62:34795
--builtin_ntp_servers=127.16.217.20:34201
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260620 20:17:09.685917 26647 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:17:09.686269 26647 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:17:09.686416 26647 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:17:09.693665 26647 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:17:09.693760 26647 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:17:09.693955 26647 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.217.1
I20260620 20:17:09.703156 26647 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.217.20:34201
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986536687152-17252-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986536687152-17252-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.217.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986536687152-17252-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.217.1
--webserver_port=0
--tserver_master_addrs=127.16.217.62:34795
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.217.1
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986536687152-17252-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:17:09.704473 26647 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:17:09.706223 26647 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:17:09.718830 26652 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:17:09.718904 26655 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:17:09.718886 26653 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:17:09.719980 26647 server_base.cc:1034] running on GCE node
I20260620 20:17:09.720554 26647 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:17:09.721470 26647 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:17:09.722689 26647 hybrid_clock.cc:648] HybridClock initialized: now 1781986629722642 us; error 63 us; skew 500 ppm
I20260620 20:17:09.722998 26647 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 20:17:09.731426 26647 webserver.cc:460] Webserver started at http://127.16.217.1:43537/ using document root <none> and password file <none>
I20260620 20:17:09.732151 26647 fs_manager.cc:359] Metadata directory not provided
I20260620 20:17:09.732298 26647 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:17:09.732627 26647 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:17:09.735649 26647 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986536687152-17252-0/minicluster-data/ts-0/data/instance:
uuid: "0a0dbe0d76a94593a2f8101a9284005d"
format_stamp: "Formatted at 2026-06-20 20:17:09 on dist-test-slave-2hpg"
server_key: "55cc2016261f4550e66cdc9898f6f83d"
server_key_iv: "c71f296a288f952e3096bf4d1bf466ec"
server_key_version: "encryptionkey@0"
I20260620 20:17:09.736449 26647 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986536687152-17252-0/minicluster-data/ts-0/wal/instance:
uuid: "0a0dbe0d76a94593a2f8101a9284005d"
format_stamp: "Formatted at 2026-06-20 20:17:09 on dist-test-slave-2hpg"
server_key: "55cc2016261f4550e66cdc9898f6f83d"
server_key_iv: "c71f296a288f952e3096bf4d1bf466ec"
server_key_version: "encryptionkey@0"
I20260620 20:17:09.741643 26647 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260620 20:17:09.745052 26661 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:17:09.746418 26647 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 20:17:09.746587 26647 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986536687152-17252-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986536687152-17252-0/minicluster-data/ts-0/wal
uuid: "0a0dbe0d76a94593a2f8101a9284005d"
format_stamp: "Formatted at 2026-06-20 20:17:09 on dist-test-slave-2hpg"
server_key: "55cc2016261f4550e66cdc9898f6f83d"
server_key_iv: "c71f296a288f952e3096bf4d1bf466ec"
server_key_version: "encryptionkey@0"
I20260620 20:17:09.746799 26647 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986536687152-17252-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986536687152-17252-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986536687152-17252-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:17:09.760234 26647 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:17:09.761068 26647 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:17:09.761381 26647 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:17:09.762379 26647 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:17:09.764045 26647 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:17:09.764317 26647 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:17:09.764463 26647 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:17:09.764559 26647 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:17:09.798367 26647 rpc_server.cc:307] RPC server started. Bound to: 127.16.217.1:39099
I20260620 20:17:09.798419 26773 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.217.1:39099 every 8 connection(s)
I20260620 20:17:09.799728 26647 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986536687152-17252-0/minicluster-data/ts-0/data/info.pb
I20260620 20:17:09.807440 17252 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu as pid 26647
I20260620 20:17:09.807614 17252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986536687152-17252-0/minicluster-data/ts-0/wal/instance
I20260620 20:17:09.808243 17252 external_mini_cluster.cc:1428] Setting key 7fe60a3c0c356f7acc46f6b2b2dcd217
I20260620 20:17:09.810956 17252 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
/tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986536687152-17252-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986536687152-17252-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986536687152-17252-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986536687152-17252-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.217.2:0
--local_ip_for_outbound_sockets=127.16.217.2
--webserver_interface=127.16.217.2
--webserver_port=0
--tserver_master_addrs=127.16.217.62:34795
--builtin_ntp_servers=127.16.217.20:34201
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260620 20:17:09.811987 26776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:09.802370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.217.1:41715 (local address 127.16.217.62:34795)
0620 20:17:09.802799 (+ 429us) server_negotiation.cc:206] Beginning negotiation
0620 20:17:09.802803 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:17:09.803941 (+ 1138us) server_negotiation.cc:407] Connection header received
0620 20:17:09.805676 (+ 1735us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:17:09.805682 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:17:09.805771 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:17:09.805910 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:17:09.806942 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:09.807757 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:09.809306 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:09.809708 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:09.809839 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:17:09.811232 (+ 1393us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:17:09.811237 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:17:09.811240 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:17:09.811255 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:17:09.811310 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:17:09.811322 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:17:09.811390 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0620 20:17:09.811672 (+ 282us) server_negotiation.cc:299] Negotiation successful
0620 20:17:09.811742 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":327,"thread_start_us":250,"threads_started":1}
I20260620 20:17:09.813457 26777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:09.802878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:34795 (local address 127.16.217.1:41715)
0620 20:17:09.803780 (+ 902us) negotiation.cc:107] Waiting for socket to connect
0620 20:17:09.803822 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 20:17:09.805381 (+ 1559us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:17:09.806140 (+ 759us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:17:09.806148 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:17:09.806232 (+ 84us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:17:09.806781 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:17:09.806797 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:17:09.807902 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:17:09.807906 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:17:09.809162 (+ 1256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:17:09.809174 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:17:09.809960 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:17:09.809968 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:17:09.810190 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:17:09.810899 (+ 709us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 20:17:09.810902 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:17:09.810937 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 20:17:09.811006 (+ 69us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:17:09.811429 (+ 423us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:17:09.811442 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:17:09.811455 (+ 13us) client_negotiation.cc:769] Sending connection context
0620 20:17:09.811652 (+ 197us) client_negotiation.cc:240] Negotiation successful
0620 20:17:09.812076 (+ 424us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":433,"thread_start_us":266,"threads_started":1}
I20260620 20:17:09.814985 26774 heartbeater.cc:347] Connected to a master server at 127.16.217.62:34795
I20260620 20:17:09.815294 26774 heartbeater.cc:464] Registering TS with master...
I20260620 20:17:09.816064 26774 heartbeater.cc:511] Master 127.16.217.62:34795 requested a full tablet report, sending...
I20260620 20:17:09.818216 26588 ts_manager.cc:194] Registered new tserver with Master: 0a0dbe0d76a94593a2f8101a9284005d (127.16.217.1:39099)
I20260620 20:17:09.819934 26588 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.217.1:41715
W20260620 20:17:10.150408 26778 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:17:10.150775 26778 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:17:10.150970 26778 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:17:10.158301 26778 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:17:10.158452 26778 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:17:10.158627 26778 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.217.2
I20260620 20:17:10.167698 26778 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.217.20:34201
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986536687152-17252-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986536687152-17252-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.217.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986536687152-17252-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.217.2
--webserver_port=0
--tserver_master_addrs=127.16.217.62:34795
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.217.2
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986536687152-17252-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:17:10.169091 26778 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:17:10.170950 26778 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:17:10.183997 26784 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:17:10.183957 26783 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:17:10.183920 26786 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:17:10.184590 26778 server_base.cc:1034] running on GCE node
I20260620 20:17:10.185256 26778 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:17:10.186177 26778 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:17:10.187373 26778 hybrid_clock.cc:648] HybridClock initialized: now 1781986630187330 us; error 31 us; skew 500 ppm
I20260620 20:17:10.187765 26778 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 20:17:10.190116 26778 webserver.cc:460] Webserver started at http://127.16.217.2:46501/ using document root <none> and password file <none>
I20260620 20:17:10.190851 26778 fs_manager.cc:359] Metadata directory not provided
I20260620 20:17:10.190980 26778 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:17:10.191305 26778 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:17:10.194306 26778 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986536687152-17252-0/minicluster-data/ts-1/data/instance:
uuid: "a0282563c22d4ba7ab68ca0ee1dfa952"
format_stamp: "Formatted at 2026-06-20 20:17:10 on dist-test-slave-2hpg"
server_key: "ebd88052924771d31833207b26f1e1b0"
server_key_iv: "4985007f17855c09e1c872de6f2dc335"
server_key_version: "encryptionkey@0"
I20260620 20:17:10.195082 26778 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986536687152-17252-0/minicluster-data/ts-1/wal/instance:
uuid: "a0282563c22d4ba7ab68ca0ee1dfa952"
format_stamp: "Formatted at 2026-06-20 20:17:10 on dist-test-slave-2hpg"
server_key: "ebd88052924771d31833207b26f1e1b0"
server_key_iv: "4985007f17855c09e1c872de6f2dc335"
server_key_version: "encryptionkey@0"
I20260620 20:17:10.200204 26778 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260620 20:17:10.203608 26792 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:17:10.204938 26778 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 20:17:10.205116 26778 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986536687152-17252-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986536687152-17252-0/minicluster-data/ts-1/wal
uuid: "a0282563c22d4ba7ab68ca0ee1dfa952"
format_stamp: "Formatted at 2026-06-20 20:17:10 on dist-test-slave-2hpg"
server_key: "ebd88052924771d31833207b26f1e1b0"
server_key_iv: "4985007f17855c09e1c872de6f2dc335"
server_key_version: "encryptionkey@0"
I20260620 20:17:10.205339 26778 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986536687152-17252-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986536687152-17252-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986536687152-17252-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:17:10.235092 26778 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:17:10.236122 26778 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:17:10.236446 26778 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:17:10.237396 26778 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:17:10.238827 26778 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:17:10.238950 26778 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:17:10.239082 26778 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:17:10.239166 26778 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:17:10.272706 26778 rpc_server.cc:307] RPC server started. Bound to: 127.16.217.2:40699
I20260620 20:17:10.272795 26904 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.217.2:40699 every 8 connection(s)
I20260620 20:17:10.274080 26778 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986536687152-17252-0/minicluster-data/ts-1/data/info.pb
I20260620 20:17:10.276476 17252 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu as pid 26778
I20260620 20:17:10.276641 17252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986536687152-17252-0/minicluster-data/ts-1/wal/instance
I20260620 20:17:10.277233 17252 external_mini_cluster.cc:1428] Setting key c1f2aa78b86d5bf932190a510cdbcb9a
I20260620 20:17:10.280292 17252 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
/tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986536687152-17252-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986536687152-17252-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986536687152-17252-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986536687152-17252-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.217.3:0
--local_ip_for_outbound_sockets=127.16.217.3
--webserver_interface=127.16.217.3
--webserver_port=0
--tserver_master_addrs=127.16.217.62:34795
--builtin_ntp_servers=127.16.217.20:34201
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260620 20:17:10.288545 26776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:10.276801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.217.2:54675 (local address 127.16.217.62:34795)
0620 20:17:10.276967 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0620 20:17:10.276971 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:17:10.279001 (+ 2030us) server_negotiation.cc:407] Connection header received
0620 20:17:10.281233 (+ 2232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:17:10.281239 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:17:10.281337 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:17:10.281474 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:17:10.282743 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:10.283527 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:10.285305 (+ 1778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:10.285926 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:10.286075 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:17:10.287503 (+ 1428us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:17:10.287513 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:17:10.287521 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:17:10.287552 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:17:10.287619 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:17:10.287639 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:17:10.287744 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0620 20:17:10.287955 (+ 211us) server_negotiation.cc:299] Negotiation successful
0620 20:17:10.288078 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 20:17:10.289366 26907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:10.277534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:34795 (local address 127.16.217.2:54675)
0620 20:17:10.278808 (+ 1274us) negotiation.cc:107] Waiting for socket to connect
0620 20:17:10.278870 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0620 20:17:10.280934 (+ 2064us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:17:10.281737 (+ 803us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:17:10.281747 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:17:10.281851 (+ 104us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:17:10.282561 (+ 710us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:17:10.282579 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:17:10.283697 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:17:10.283704 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:17:10.285127 (+ 1423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:17:10.285150 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:17:10.286154 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:17:10.286160 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:17:10.286377 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:17:10.287153 (+ 776us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 20:17:10.287156 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:17:10.287199 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 20:17:10.287289 (+ 90us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:17:10.287747 (+ 458us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:17:10.287755 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:17:10.287761 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 20:17:10.287912 (+ 151us) client_negotiation.cc:240] Negotiation successful
0620 20:17:10.288239 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":665,"thread_start_us":400,"threads_started":1}
I20260620 20:17:10.290917 26905 heartbeater.cc:347] Connected to a master server at 127.16.217.62:34795
I20260620 20:17:10.291271 26905 heartbeater.cc:464] Registering TS with master...
I20260620 20:17:10.292377 26905 heartbeater.cc:511] Master 127.16.217.62:34795 requested a full tablet report, sending...
I20260620 20:17:10.294150 26588 ts_manager.cc:194] Registered new tserver with Master: a0282563c22d4ba7ab68ca0ee1dfa952 (127.16.217.2:40699)
I20260620 20:17:10.295562 26588 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.217.2:54675
W20260620 20:17:10.610110 26908 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:17:10.610374 26908 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:17:10.610452 26908 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:17:10.617509 26908 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:17:10.617616 26908 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:17:10.617717 26908 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.217.3
I20260620 20:17:10.626992 26908 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.217.20:34201
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986536687152-17252-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986536687152-17252-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.217.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986536687152-17252-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.217.3
--webserver_port=0
--tserver_master_addrs=127.16.217.62:34795
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.217.3
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986536687152-17252-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:17:10.628232 26908 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:17:10.629972 26908 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:17:10.642642 26916 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:17:10.642692 26914 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:17:10.643348 26913 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:17:10.643374 26908 server_base.cc:1034] running on GCE node
I20260620 20:17:10.644006 26908 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:17:10.644971 26908 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:17:10.646190 26908 hybrid_clock.cc:648] HybridClock initialized: now 1781986630646142 us; error 74 us; skew 500 ppm
I20260620 20:17:10.646518 26908 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 20:17:10.648685 26908 webserver.cc:460] Webserver started at http://127.16.217.3:43219/ using document root <none> and password file <none>
I20260620 20:17:10.649380 26908 fs_manager.cc:359] Metadata directory not provided
I20260620 20:17:10.649508 26908 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:17:10.649842 26908 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:17:10.652685 26908 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986536687152-17252-0/minicluster-data/ts-2/data/instance:
uuid: "2ef8e30fc823484682fdc70117f275cd"
format_stamp: "Formatted at 2026-06-20 20:17:10 on dist-test-slave-2hpg"
server_key: "e459b10d2b6a934b371a3e1bfedf7f54"
server_key_iv: "82f5b4c347ada46b5c398400d3ee2bdd"
server_key_version: "encryptionkey@0"
I20260620 20:17:10.653450 26908 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986536687152-17252-0/minicluster-data/ts-2/wal/instance:
uuid: "2ef8e30fc823484682fdc70117f275cd"
format_stamp: "Formatted at 2026-06-20 20:17:10 on dist-test-slave-2hpg"
server_key: "e459b10d2b6a934b371a3e1bfedf7f54"
server_key_iv: "82f5b4c347ada46b5c398400d3ee2bdd"
server_key_version: "encryptionkey@0"
I20260620 20:17:10.658439 26908 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.002s
I20260620 20:17:10.661636 26922 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:17:10.663200 26908 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 20:17:10.663403 26908 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986536687152-17252-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986536687152-17252-0/minicluster-data/ts-2/wal
uuid: "2ef8e30fc823484682fdc70117f275cd"
format_stamp: "Formatted at 2026-06-20 20:17:10 on dist-test-slave-2hpg"
server_key: "e459b10d2b6a934b371a3e1bfedf7f54"
server_key_iv: "82f5b4c347ada46b5c398400d3ee2bdd"
server_key_version: "encryptionkey@0"
I20260620 20:17:10.663650 26908 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986536687152-17252-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986536687152-17252-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986536687152-17252-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:17:10.696946 26908 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:17:10.697844 26908 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:17:10.698225 26908 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:17:10.699213 26908 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:17:10.700820 26908 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:17:10.700906 26908 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:17:10.701049 26908 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:17:10.701146 26908 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:17:10.734691 26908 rpc_server.cc:307] RPC server started. Bound to: 127.16.217.3:40197
I20260620 20:17:10.734866 27034 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.217.3:40197 every 8 connection(s)
I20260620 20:17:10.736027 26908 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986536687152-17252-0/minicluster-data/ts-2/data/info.pb
I20260620 20:17:10.746348 17252 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu as pid 26908
I20260620 20:17:10.746537 17252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986536687152-17252-0/minicluster-data/ts-2/wal/instance
I20260620 20:17:10.747198 17252 external_mini_cluster.cc:1428] Setting key ce739b270140b9611d301431d4f5557e
I20260620 20:17:10.749195 26776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:10.738861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.217.3:41333 (local address 127.16.217.62:34795)
0620 20:17:10.739068 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0620 20:17:10.739074 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:17:10.740560 (+ 1486us) server_negotiation.cc:407] Connection header received
0620 20:17:10.742526 (+ 1966us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:17:10.742530 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:17:10.742594 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:17:10.742682 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:17:10.743757 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:10.744460 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:10.746141 (+ 1681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:10.746600 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:10.746762 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:17:10.748277 (+ 1515us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:17:10.748283 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:17:10.748295 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:17:10.748315 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:17:10.748351 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:17:10.748364 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:17:10.748479 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0620 20:17:10.748729 (+ 250us) server_negotiation.cc:299] Negotiation successful
0620 20:17:10.748832 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260620 20:17:10.750211 27037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:10.739449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:34795 (local address 127.16.217.3:41333)
0620 20:17:10.740403 (+ 954us) negotiation.cc:107] Waiting for socket to connect
0620 20:17:10.740440 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 20:17:10.742239 (+ 1799us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:17:10.742896 (+ 657us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:17:10.742905 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:17:10.742987 (+ 82us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:17:10.743585 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:17:10.743602 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:17:10.744609 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:17:10.744613 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:17:10.745990 (+ 1377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:17:10.746008 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:17:10.746855 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:17:10.746863 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:17:10.747198 (+ 335us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:17:10.747991 (+ 793us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 20:17:10.747994 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:17:10.748029 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 20:17:10.748099 (+ 70us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:17:10.748479 (+ 380us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:17:10.748487 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:17:10.748493 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 20:17:10.748697 (+ 204us) client_negotiation.cc:240] Negotiation successful
0620 20:17:10.748998 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":506,"thread_start_us":320,"threads_started":1}
I20260620 20:17:10.751695 27035 heartbeater.cc:347] Connected to a master server at 127.16.217.62:34795
I20260620 20:17:10.752022 27035 heartbeater.cc:464] Registering TS with master...
I20260620 20:17:10.752805 27035 heartbeater.cc:511] Master 127.16.217.62:34795 requested a full tablet report, sending...
I20260620 20:17:10.754216 26588 ts_manager.cc:194] Registered new tserver with Master: 2ef8e30fc823484682fdc70117f275cd (127.16.217.3:40197)
I20260620 20:17:10.755120 26588 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.217.3:41333
I20260620 20:17:10.762876 17252 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260620 20:17:10.768989 26776 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:17:10.767239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51736 (local address 127.16.217.62:34795)
0620 20:17:10.767434 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0620 20:17:10.767442 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 20:17:10.767948 (+ 506us) server_negotiation.cc:407] Connection header received
0620 20:17:10.768141 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:17:10.768146 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:17:10.768231 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:17:10.768335 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:17:10.768724 (+ 389us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51736: BlockingRecv error: recv got EOF from 127.0.0.1:51736 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 20:17:10.769555 17252 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu pid 26647
I20260620 20:17:10.777487 27044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:10.770481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54062 (local address 127.16.217.1:39099)
0620 20:17:10.770937 (+ 456us) server_negotiation.cc:206] Beginning negotiation
0620 20:17:10.770943 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:17:10.770968 (+ 25us) server_negotiation.cc:407] Connection header received
0620 20:17:10.771020 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:17:10.771026 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:17:10.771299 (+ 273us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:17:10.771463 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:17:10.771873 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:10.773037 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:10.774380 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:10.774930 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:10.775063 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:17:10.776107 (+ 1044us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:17:10.776116 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:17:10.776134 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:17:10.776719 (+ 585us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:17:10.776765 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:17:10.776785 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:17:10.776916 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0620 20:17:10.777029 (+ 113us) server_negotiation.cc:299] Negotiation successful
0620 20:17:10.777163 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":331,"thread_start_us":223,"threads_started":1}
I20260620 20:17:10.778777 26769 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:17:10.858541 26774 heartbeater.cc:503] Master 127.16.217.62:34795 was elected leader, sending a full tablet report...
I20260620 20:17:10.860097 17252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu with pid 26647
W20260620 20:17:10.876641 26573 connection.cc:535] server connection from 127.16.217.1:41715 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 20:17:10.876995 17252 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu pid 26778
I20260620 20:17:10.884773 27046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:10.877965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51792 (local address 127.16.217.2:40699)
0620 20:17:10.878564 (+ 599us) server_negotiation.cc:206] Beginning negotiation
0620 20:17:10.878572 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 20:17:10.878599 (+ 27us) server_negotiation.cc:407] Connection header received
0620 20:17:10.878665 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:17:10.878671 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:17:10.878902 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:17:10.879034 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:17:10.879382 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:10.880404 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:10.881635 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:10.882281 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:10.882379 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:17:10.883342 (+ 963us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:17:10.883350 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:17:10.883365 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:17:10.883958 (+ 593us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:17:10.884002 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:17:10.884031 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:17:10.884117 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0620 20:17:10.884260 (+ 143us) server_negotiation.cc:299] Negotiation successful
0620 20:17:10.884400 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":435,"thread_start_us":301,"threads_started":1}
I20260620 20:17:10.886161 26900 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:17:10.967146 17252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu with pid 26778
I20260620 20:17:10.985128 17252 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu pid 26908
I20260620 20:17:10.993439 27048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:10.986259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56910 (local address 127.16.217.3:40197)
0620 20:17:10.986930 (+ 671us) server_negotiation.cc:206] Beginning negotiation
0620 20:17:10.986937 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:17:10.986966 (+ 29us) server_negotiation.cc:407] Connection header received
0620 20:17:10.987028 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:17:10.987035 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:17:10.987274 (+ 239us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:17:10.987401 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:17:10.987963 (+ 562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:10.989035 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:10.990373 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:10.990896 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:10.990993 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:17:10.991939 (+ 946us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:17:10.991947 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:17:10.991960 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:17:10.992575 (+ 615us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:17:10.992662 (+ 87us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:17:10.992688 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:17:10.992783 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0620 20:17:10.992927 (+ 144us) server_negotiation.cc:299] Negotiation successful
0620 20:17:10.993031 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":537,"thread_start_us":387,"threads_started":1}
I20260620 20:17:10.994776 27030 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:17:11.076010 17252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu with pid 26908
I20260620 20:17:11.091889 17252 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu pid 26557
I20260620 20:17:11.093192 26618 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:17:11.197741 17252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu with pid 26557
2026-06-20T20:17:11Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (2442 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-06-20T20:17:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T20:17:11Z Disabled control of system clock
I20260620 20:17:11.272025 17252 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
/tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986536687152-17252-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986536687152-17252-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986536687152-17252-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986536687152-17252-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.217.62:42933
--webserver_interface=127.16.217.62
--webserver_port=0
--builtin_ntp_servers=127.16.217.20:36501
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.217.62:42933
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260620 20:17:11.610141 27054 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:17:11.610453 27054 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:17:11.610613 27054 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:17:11.618255 27054 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:17:11.618371 27054 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:17:11.618477 27054 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:17:11.618606 27054 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:17:11.618686 27054 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:17:11.628767 27054 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.217.20:36501
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986536687152-17252-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986536687152-17252-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.217.62:42933
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.217.62:42933
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986536687152-17252-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.217.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986536687152-17252-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:17:11.630256 27054 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:17:11.632212 27054 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:17:11.644924 27060 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:17:11.645030 27059 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:17:11.645058 27062 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:17:11.646008 27054 server_base.cc:1034] running on GCE node
I20260620 20:17:11.646875 27054 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:17:11.648306 27054 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:17:11.649518 27054 hybrid_clock.cc:648] HybridClock initialized: now 1781986631649486 us; error 39 us; skew 500 ppm
I20260620 20:17:11.649839 27054 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 20:17:11.652112 27054 webserver.cc:460] Webserver started at http://127.16.217.62:39571/ using document root <none> and password file <none>
I20260620 20:17:11.652781 27054 fs_manager.cc:359] Metadata directory not provided
I20260620 20:17:11.652868 27054 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:17:11.653123 27054 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:17:11.656082 27054 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986536687152-17252-0/minicluster-data/master-0/data/instance:
uuid: "6d86d7929c1846d88b45a645b4bebe65"
format_stamp: "Formatted at 2026-06-20 20:17:11 on dist-test-slave-2hpg"
server_key: "f8bcfe34c6b548d9a46000ece17df000"
server_key_iv: "f65fe7a060389e8256b816d10b7a4655"
server_key_version: "encryptionkey@0"
I20260620 20:17:11.656813 27054 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986536687152-17252-0/minicluster-data/master-0/wal/instance:
uuid: "6d86d7929c1846d88b45a645b4bebe65"
format_stamp: "Formatted at 2026-06-20 20:17:11 on dist-test-slave-2hpg"
server_key: "f8bcfe34c6b548d9a46000ece17df000"
server_key_iv: "f65fe7a060389e8256b816d10b7a4655"
server_key_version: "encryptionkey@0"
I20260620 20:17:11.661837 27054 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260620 20:17:11.665493 27068 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:17:11.666908 27054 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 20:17:11.667079 27054 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986536687152-17252-0/minicluster-data/master-0/data,/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986536687152-17252-0/minicluster-data/master-0/wal
uuid: "6d86d7929c1846d88b45a645b4bebe65"
format_stamp: "Formatted at 2026-06-20 20:17:11 on dist-test-slave-2hpg"
server_key: "f8bcfe34c6b548d9a46000ece17df000"
server_key_iv: "f65fe7a060389e8256b816d10b7a4655"
server_key_version: "encryptionkey@0"
I20260620 20:17:11.667274 27054 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986536687152-17252-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986536687152-17252-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986536687152-17252-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:17:11.672973 27054 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:17:11.673776 27054 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:17:11.674073 27054 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:17:11.692261 27054 rpc_server.cc:307] RPC server started. Bound to: 127.16.217.62:42933
I20260620 20:17:11.692310 27119 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.217.62:42933 every 8 connection(s)
I20260620 20:17:11.693589 27054 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986536687152-17252-0/minicluster-data/master-0/data/info.pb
I20260620 20:17:11.694375 17252 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu as pid 27054
I20260620 20:17:11.694583 17252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986536687152-17252-0/minicluster-data/master-0/wal/instance
I20260620 20:17:11.695295 17252 external_mini_cluster.cc:1428] Setting key d296d41eec9f62f38e4a2ac6cb57da2a
I20260620 20:17:11.698143 27120 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:17:11.703508 27123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:11.697333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57594 (local address 127.16.217.62:42933)
0620 20:17:11.698385 (+ 1052us) server_negotiation.cc:206] Beginning negotiation
0620 20:17:11.698396 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 20:17:11.698439 (+ 43us) server_negotiation.cc:407] Connection header received
0620 20:17:11.699981 (+ 1542us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:17:11.700001 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:17:11.700399 (+ 398us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:17:11.700724 (+ 325us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:17:11.701840 (+ 1116us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:17:11.701846 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:17:11.701913 (+ 67us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:17:11.701936 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:17:11.702035 (+ 99us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:17:11.702059 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:17:11.702291 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0620 20:17:11.702427 (+ 136us) server_negotiation.cc:299] Negotiation successful
0620 20:17:11.702649 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":587,"thread_start_us":370,"threads_started":1}
I20260620 20:17:11.709049 27120 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6d86d7929c1846d88b45a645b4bebe65: Bootstrap starting.
I20260620 20:17:11.712020 27120 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6d86d7929c1846d88b45a645b4bebe65: Neither blocks nor log segments found. Creating new log.
I20260620 20:17:11.712993 27120 log.cc:824] T 00000000000000000000000000000000 P 6d86d7929c1846d88b45a645b4bebe65: Log is configured to *not* fsync() on all Append() calls
I20260620 20:17:11.715762 27120 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6d86d7929c1846d88b45a645b4bebe65: No bootstrap required, opened a new log
I20260620 20:17:11.722995 27120 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6d86d7929c1846d88b45a645b4bebe65 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d86d7929c1846d88b45a645b4bebe65" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 42933 } }
I20260620 20:17:11.723393 27120 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6d86d7929c1846d88b45a645b4bebe65 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:17:11.723505 27120 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6d86d7929c1846d88b45a645b4bebe65 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d86d7929c1846d88b45a645b4bebe65, State: Initialized, Role: FOLLOWER
I20260620 20:17:11.724074 27120 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6d86d7929c1846d88b45a645b4bebe65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d86d7929c1846d88b45a645b4bebe65" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 42933 } }
I20260620 20:17:11.724475 27120 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6d86d7929c1846d88b45a645b4bebe65 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 20:17:11.724757 27120 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6d86d7929c1846d88b45a645b4bebe65 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 20:17:11.724962 27120 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6d86d7929c1846d88b45a645b4bebe65 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:17:11.727245 27120 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6d86d7929c1846d88b45a645b4bebe65 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d86d7929c1846d88b45a645b4bebe65" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 42933 } }
I20260620 20:17:11.727802 27120 leader_election.cc:302] T 00000000000000000000000000000000 P 6d86d7929c1846d88b45a645b4bebe65 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6d86d7929c1846d88b45a645b4bebe65; no voters:
I20260620 20:17:11.728384 27120 leader_election.cc:288] T 00000000000000000000000000000000 P 6d86d7929c1846d88b45a645b4bebe65 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 20:17:11.728510 27125 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6d86d7929c1846d88b45a645b4bebe65 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:17:11.729242 27125 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6d86d7929c1846d88b45a645b4bebe65 [term 1 LEADER]: Becoming Leader. State: Replica: 6d86d7929c1846d88b45a645b4bebe65, State: Running, Role: LEADER
I20260620 20:17:11.729764 27125 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6d86d7929c1846d88b45a645b4bebe65 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d86d7929c1846d88b45a645b4bebe65" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 42933 } }
I20260620 20:17:11.730645 27120 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6d86d7929c1846d88b45a645b4bebe65 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:17:11.732373 27126 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d86d7929c1846d88b45a645b4bebe65 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6d86d7929c1846d88b45a645b4bebe65" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d86d7929c1846d88b45a645b4bebe65" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 42933 } } }
I20260620 20:17:11.732731 27126 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d86d7929c1846d88b45a645b4bebe65 [sys.catalog]: This master's current role is: LEADER
I20260620 20:17:11.732949 27127 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d86d7929c1846d88b45a645b4bebe65 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6d86d7929c1846d88b45a645b4bebe65. Latest consensus state: current_term: 1 leader_uuid: "6d86d7929c1846d88b45a645b4bebe65" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d86d7929c1846d88b45a645b4bebe65" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 42933 } } }
I20260620 20:17:11.733287 27127 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d86d7929c1846d88b45a645b4bebe65 [sys.catalog]: This master's current role is: LEADER
I20260620 20:17:11.735823 27130 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 20:17:11.741252 27130 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 20:17:11.749598 27130 catalog_manager.cc:1347] Generated new cluster ID: 34e8f2cd35b0470cbb0b1613faa51c94
I20260620 20:17:11.749794 27130 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 20:17:11.781533 27130 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 20:17:11.782061 27130 catalog_manager.cc:1504] Loading token signing keys...
I20260620 20:17:11.787954 27130 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6d86d7929c1846d88b45a645b4bebe65: Generated new TSK 0
I20260620 20:17:11.788760 27130 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 20:17:11.800813 17252 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
/tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986536687152-17252-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986536687152-17252-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986536687152-17252-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986536687152-17252-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.217.1:0
--local_ip_for_outbound_sockets=127.16.217.1
--webserver_interface=127.16.217.1
--webserver_port=0
--tserver_master_addrs=127.16.217.62:42933
--builtin_ntp_servers=127.16.217.20:36501
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260620 20:17:12.150669 27144 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:17:12.150955 27144 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:17:12.151120 27144 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:17:12.158468 27144 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:17:12.158561 27144 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:17:12.158741 27144 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.217.1
I20260620 20:17:12.168085 27144 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.217.20:36501
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986536687152-17252-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986536687152-17252-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.217.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986536687152-17252-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.217.1
--webserver_port=0
--tserver_master_addrs=127.16.217.62:42933
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.217.1
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986536687152-17252-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:17:12.169365 27144 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:17:12.171285 27144 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:17:12.184487 27150 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:17:12.184548 27152 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:17:12.184579 27149 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:17:12.185212 27144 server_base.cc:1034] running on GCE node
I20260620 20:17:12.185786 27144 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:17:12.186733 27144 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:17:12.187959 27144 hybrid_clock.cc:648] HybridClock initialized: now 1781986632187913 us; error 37 us; skew 500 ppm
I20260620 20:17:12.188280 27144 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 20:17:12.190514 27144 webserver.cc:460] Webserver started at http://127.16.217.1:46327/ using document root <none> and password file <none>
I20260620 20:17:12.191200 27144 fs_manager.cc:359] Metadata directory not provided
I20260620 20:17:12.191326 27144 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:17:12.191630 27144 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:17:12.196000 27144 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986536687152-17252-0/minicluster-data/ts-0/data/instance:
uuid: "1a5398d64dd7409895f0835171b66a44"
format_stamp: "Formatted at 2026-06-20 20:17:12 on dist-test-slave-2hpg"
server_key: "5bd7d938ce2fafecde30edd9dc0528dc"
server_key_iv: "5930cd7a9a5443bd6e8c8ae884d47444"
server_key_version: "encryptionkey@0"
I20260620 20:17:12.196766 27144 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986536687152-17252-0/minicluster-data/ts-0/wal/instance:
uuid: "1a5398d64dd7409895f0835171b66a44"
format_stamp: "Formatted at 2026-06-20 20:17:12 on dist-test-slave-2hpg"
server_key: "5bd7d938ce2fafecde30edd9dc0528dc"
server_key_iv: "5930cd7a9a5443bd6e8c8ae884d47444"
server_key_version: "encryptionkey@0"
I20260620 20:17:12.201758 27144 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.007s
I20260620 20:17:12.205276 27158 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:17:12.206678 27144 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260620 20:17:12.206852 27144 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986536687152-17252-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986536687152-17252-0/minicluster-data/ts-0/wal
uuid: "1a5398d64dd7409895f0835171b66a44"
format_stamp: "Formatted at 2026-06-20 20:17:12 on dist-test-slave-2hpg"
server_key: "5bd7d938ce2fafecde30edd9dc0528dc"
server_key_iv: "5930cd7a9a5443bd6e8c8ae884d47444"
server_key_version: "encryptionkey@0"
I20260620 20:17:12.207063 27144 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986536687152-17252-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986536687152-17252-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986536687152-17252-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:17:12.212659 27144 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:17:12.213393 27144 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:17:12.213653 27144 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:17:12.214654 27144 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:17:12.215981 27144 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:17:12.216104 27144 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:17:12.216220 27144 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:17:12.216316 27144 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:17:12.249282 27144 rpc_server.cc:307] RPC server started. Bound to: 127.16.217.1:42605
I20260620 20:17:12.249369 27270 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.217.1:42605 every 8 connection(s)
I20260620 20:17:12.250620 27144 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986536687152-17252-0/minicluster-data/ts-0/data/info.pb
I20260620 20:17:12.257393 17252 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu as pid 27144
I20260620 20:17:12.257638 17252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986536687152-17252-0/minicluster-data/ts-0/wal/instance
I20260620 20:17:12.258489 17252 external_mini_cluster.cc:1428] Setting key 71fdf312e40585c6f41ac7f3f62f02f6
I20260620 20:17:12.258630 27273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:12.253031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.217.1:37023 (local address 127.16.217.62:42933)
0620 20:17:12.253450 (+ 419us) server_negotiation.cc:206] Beginning negotiation
0620 20:17:12.253454 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:17:12.254547 (+ 1093us) server_negotiation.cc:407] Connection header received
0620 20:17:12.256173 (+ 1626us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:17:12.256178 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:17:12.256255 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:17:12.256381 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:17:12.257768 (+ 1387us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:17:12.257775 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:17:12.257779 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:17:12.257802 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:17:12.257860 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:17:12.257909 (+ 49us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:17:12.258053 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0620 20:17:12.258234 (+ 181us) server_negotiation.cc:299] Negotiation successful
0620 20:17:12.258305 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":320,"thread_start_us":233,"threads_started":1}
I20260620 20:17:12.259703 27274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:12.253511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:42933 (local address 127.16.217.1:37023)
0620 20:17:12.254409 (+ 898us) negotiation.cc:107] Waiting for socket to connect
0620 20:17:12.254442 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0620 20:17:12.255915 (+ 1473us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:17:12.256604 (+ 689us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:17:12.256612 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:17:12.256698 (+ 86us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:17:12.257371 (+ 673us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 20:17:12.257374 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:17:12.257409 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 20:17:12.257486 (+ 77us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:17:12.258019 (+ 533us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:17:12.258028 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:17:12.258034 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 20:17:12.258264 (+ 230us) client_negotiation.cc:240] Negotiation successful
0620 20:17:12.258450 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":466,"thread_start_us":247,"threads_started":1}
I20260620 20:17:12.261238 27271 heartbeater.cc:347] Connected to a master server at 127.16.217.62:42933
I20260620 20:17:12.261497 17252 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
/tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986536687152-17252-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986536687152-17252-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986536687152-17252-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986536687152-17252-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.217.2:0
--local_ip_for_outbound_sockets=127.16.217.2
--webserver_interface=127.16.217.2
--webserver_port=0
--tserver_master_addrs=127.16.217.62:42933
--builtin_ntp_servers=127.16.217.20:36501
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260620 20:17:12.261524 27271 heartbeater.cc:464] Registering TS with master...
I20260620 20:17:12.262213 27271 heartbeater.cc:511] Master 127.16.217.62:42933 requested a full tablet report, sending...
I20260620 20:17:12.264169 27085 ts_manager.cc:194] Registered new tserver with Master: 1a5398d64dd7409895f0835171b66a44 (127.16.217.1:42605)
W20260620 20:17:12.592192 27275 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:17:12.592485 27275 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:17:12.592624 27275 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:17:12.600031 27275 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:17:12.600145 27275 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:17:12.600312 27275 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.217.2
I20260620 20:17:12.609994 27275 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.217.20:36501
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986536687152-17252-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986536687152-17252-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.217.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986536687152-17252-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.217.2
--webserver_port=0
--tserver_master_addrs=127.16.217.62:42933
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.217.2
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986536687152-17252-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:17:12.611481 27275 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:17:12.613427 27275 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:17:12.627067 27281 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:17:12.627094 27283 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:17:12.627094 27280 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:17:12.627987 27275 server_base.cc:1034] running on GCE node
I20260620 20:17:12.628597 27275 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:17:12.629555 27275 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:17:12.630779 27275 hybrid_clock.cc:648] HybridClock initialized: now 1781986632630733 us; error 39 us; skew 500 ppm
I20260620 20:17:12.631126 27275 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 20:17:12.633400 27275 webserver.cc:460] Webserver started at http://127.16.217.2:45849/ using document root <none> and password file <none>
I20260620 20:17:12.634199 27275 fs_manager.cc:359] Metadata directory not provided
I20260620 20:17:12.634320 27275 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:17:12.634672 27275 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:17:12.637481 27275 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986536687152-17252-0/minicluster-data/ts-1/data/instance:
uuid: "de52a92e0d5647bca09d39180977ac99"
format_stamp: "Formatted at 2026-06-20 20:17:12 on dist-test-slave-2hpg"
server_key: "4010a929953144c2cdaeacaa2e5daabc"
server_key_iv: "ce341e44c63692f15f3432c3915408df"
server_key_version: "encryptionkey@0"
I20260620 20:17:12.638345 27275 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986536687152-17252-0/minicluster-data/ts-1/wal/instance:
uuid: "de52a92e0d5647bca09d39180977ac99"
format_stamp: "Formatted at 2026-06-20 20:17:12 on dist-test-slave-2hpg"
server_key: "4010a929953144c2cdaeacaa2e5daabc"
server_key_iv: "ce341e44c63692f15f3432c3915408df"
server_key_version: "encryptionkey@0"
I20260620 20:17:12.643743 27275 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 20:17:12.647360 27289 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:17:12.648715 27275 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260620 20:17:12.648895 27275 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986536687152-17252-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986536687152-17252-0/minicluster-data/ts-1/wal
uuid: "de52a92e0d5647bca09d39180977ac99"
format_stamp: "Formatted at 2026-06-20 20:17:12 on dist-test-slave-2hpg"
server_key: "4010a929953144c2cdaeacaa2e5daabc"
server_key_iv: "ce341e44c63692f15f3432c3915408df"
server_key_version: "encryptionkey@0"
I20260620 20:17:12.649101 27275 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986536687152-17252-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986536687152-17252-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986536687152-17252-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:17:12.654706 27275 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:17:12.655493 27275 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:17:12.655762 27275 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:17:12.656643 27275 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:17:12.658083 27275 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:17:12.658192 27275 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:17:12.658314 27275 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:17:12.658403 27275 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:17:12.690743 27275 rpc_server.cc:307] RPC server started. Bound to: 127.16.217.2:38939
I20260620 20:17:12.690832 27401 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.217.2:38939 every 8 connection(s)
I20260620 20:17:12.692046 27275 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986536687152-17252-0/minicluster-data/ts-1/data/info.pb
I20260620 20:17:12.694005 17252 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu as pid 27275
I20260620 20:17:12.694198 17252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986536687152-17252-0/minicluster-data/ts-1/wal/instance
I20260620 20:17:12.694772 17252 external_mini_cluster.cc:1428] Setting key 6a3a8303bf1b6ee8e784868004778096
I20260620 20:17:12.697535 17252 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
/tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986536687152-17252-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986536687152-17252-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986536687152-17252-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986536687152-17252-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.217.3:0
--local_ip_for_outbound_sockets=127.16.217.3
--webserver_interface=127.16.217.3
--webserver_port=0
--tserver_master_addrs=127.16.217.62:42933
--builtin_ntp_servers=127.16.217.20:36501
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260620 20:17:12.701701 27273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:12.694796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.217.2:46521 (local address 127.16.217.62:42933)
0620 20:17:12.694985 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0620 20:17:12.694991 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:17:12.696618 (+ 1627us) server_negotiation.cc:407] Connection header received
0620 20:17:12.699159 (+ 2541us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:17:12.699176 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:17:12.699258 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:17:12.699363 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:17:12.700863 (+ 1500us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:17:12.700871 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:17:12.700879 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:17:12.700900 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:17:12.700948 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:17:12.700966 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:17:12.701083 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0620 20:17:12.701281 (+ 198us) server_negotiation.cc:299] Negotiation successful
0620 20:17:12.701347 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260620 20:17:12.702839 27404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:12.695427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:42933 (local address 127.16.217.2:46521)
0620 20:17:12.696469 (+ 1042us) negotiation.cc:107] Waiting for socket to connect
0620 20:17:12.696514 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 20:17:12.698865 (+ 2351us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:17:12.699594 (+ 729us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:17:12.699604 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:17:12.699719 (+ 115us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:17:12.700492 (+ 773us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 20:17:12.700496 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:17:12.700547 (+ 51us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 20:17:12.700662 (+ 115us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:17:12.701067 (+ 405us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:17:12.701076 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:17:12.701084 (+ 8us) client_negotiation.cc:769] Sending connection context
0620 20:17:12.701330 (+ 246us) client_negotiation.cc:240] Negotiation successful
0620 20:17:12.701569 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":514,"thread_start_us":763,"threads_started":1}
I20260620 20:17:12.704164 27402 heartbeater.cc:347] Connected to a master server at 127.16.217.62:42933
I20260620 20:17:12.704440 27402 heartbeater.cc:464] Registering TS with master...
I20260620 20:17:12.705072 27402 heartbeater.cc:511] Master 127.16.217.62:42933 requested a full tablet report, sending...
I20260620 20:17:12.706562 27085 ts_manager.cc:194] Registered new tserver with Master: de52a92e0d5647bca09d39180977ac99 (127.16.217.2:38939)
W20260620 20:17:13.018847 27405 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:17:13.019129 27405 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:17:13.019285 27405 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:17:13.026486 27405 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:17:13.026584 27405 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:17:13.026762 27405 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.217.3
I20260620 20:17:13.035749 27405 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.217.20:36501
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986536687152-17252-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986536687152-17252-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.217.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986536687152-17252-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.217.3
--webserver_port=0
--tserver_master_addrs=127.16.217.62:42933
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.217.3
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986536687152-17252-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:17:13.037101 27405 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:17:13.038836 27405 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:17:13.051020 27411 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:17:13.050981 27410 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:17:13.051870 27413 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:17:13.053109 27405 server_base.cc:1034] running on GCE node
I20260620 20:17:13.053721 27405 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:17:13.054666 27405 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:17:13.055927 27405 hybrid_clock.cc:648] HybridClock initialized: now 1781986633055874 us; error 40 us; skew 500 ppm
I20260620 20:17:13.056281 27405 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 20:17:13.058503 27405 webserver.cc:460] Webserver started at http://127.16.217.3:42219/ using document root <none> and password file <none>
I20260620 20:17:13.059190 27405 fs_manager.cc:359] Metadata directory not provided
I20260620 20:17:13.059319 27405 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:17:13.059628 27405 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:17:13.062529 27405 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986536687152-17252-0/minicluster-data/ts-2/data/instance:
uuid: "eb64735a7d8a42a2aa0a97e37dfd0ec8"
format_stamp: "Formatted at 2026-06-20 20:17:13 on dist-test-slave-2hpg"
server_key: "4fff4b98c6689d298eab3dc9911c20da"
server_key_iv: "54b67b25db80e4ba9f90238aa4fe5443"
server_key_version: "encryptionkey@0"
I20260620 20:17:13.063306 27405 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986536687152-17252-0/minicluster-data/ts-2/wal/instance:
uuid: "eb64735a7d8a42a2aa0a97e37dfd0ec8"
format_stamp: "Formatted at 2026-06-20 20:17:13 on dist-test-slave-2hpg"
server_key: "4fff4b98c6689d298eab3dc9911c20da"
server_key_iv: "54b67b25db80e4ba9f90238aa4fe5443"
server_key_version: "encryptionkey@0"
I20260620 20:17:13.068203 27405 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260620 20:17:13.071686 27419 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:17:13.073091 27405 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 20:17:13.073268 27405 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986536687152-17252-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986536687152-17252-0/minicluster-data/ts-2/wal
uuid: "eb64735a7d8a42a2aa0a97e37dfd0ec8"
format_stamp: "Formatted at 2026-06-20 20:17:13 on dist-test-slave-2hpg"
server_key: "4fff4b98c6689d298eab3dc9911c20da"
server_key_iv: "54b67b25db80e4ba9f90238aa4fe5443"
server_key_version: "encryptionkey@0"
I20260620 20:17:13.073472 27405 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986536687152-17252-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986536687152-17252-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986536687152-17252-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:17:13.078871 27405 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:17:13.079620 27405 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:17:13.079883 27405 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:17:13.080857 27405 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:17:13.082237 27405 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:17:13.082343 27405 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:17:13.082466 27405 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:17:13.082554 27405 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:17:13.115767 27405 rpc_server.cc:307] RPC server started. Bound to: 127.16.217.3:35719
I20260620 20:17:13.115829 27531 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.217.3:35719 every 8 connection(s)
I20260620 20:17:13.117409 27405 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986536687152-17252-0/minicluster-data/ts-2/data/info.pb
I20260620 20:17:13.121533 17252 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu as pid 27405
I20260620 20:17:13.121714 17252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986536687152-17252-0/minicluster-data/ts-2/wal/instance
I20260620 20:17:13.122222 17252 external_mini_cluster.cc:1428] Setting key 65d561b2ec42b703a48117e3bb360af0
I20260620 20:17:13.125751 27273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:13.119995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.217.3:40409 (local address 127.16.217.62:42933)
0620 20:17:13.120173 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0620 20:17:13.120178 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:17:13.121598 (+ 1420us) server_negotiation.cc:407] Connection header received
0620 20:17:13.123610 (+ 2012us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:17:13.123615 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:17:13.123691 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:17:13.123807 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:17:13.125177 (+ 1370us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:17:13.125182 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:17:13.125185 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:17:13.125201 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:17:13.125231 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:17:13.125242 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:17:13.125329 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0620 20:17:13.125499 (+ 170us) server_negotiation.cc:299] Negotiation successful
0620 20:17:13.125545 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 20:17:13.126713 27534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:13.120546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:42933 (local address 127.16.217.3:40409)
0620 20:17:13.121445 (+ 899us) negotiation.cc:107] Waiting for socket to connect
0620 20:17:13.121485 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 20:17:13.123344 (+ 1859us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:17:13.124031 (+ 687us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:17:13.124039 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:17:13.124130 (+ 91us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:17:13.124829 (+ 699us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 20:17:13.124831 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:17:13.124868 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 20:17:13.124998 (+ 130us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:17:13.125327 (+ 329us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:17:13.125334 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:17:13.125339 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 20:17:13.125546 (+ 207us) client_negotiation.cc:240] Negotiation successful
0620 20:17:13.125767 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":445,"thread_start_us":268,"threads_started":1}
I20260620 20:17:13.127983 27532 heartbeater.cc:347] Connected to a master server at 127.16.217.62:42933
I20260620 20:17:13.128283 27532 heartbeater.cc:464] Registering TS with master...
I20260620 20:17:13.128831 27532 heartbeater.cc:511] Master 127.16.217.62:42933 requested a full tablet report, sending...
I20260620 20:17:13.130152 27085 ts_manager.cc:194] Registered new tserver with Master: eb64735a7d8a42a2aa0a97e37dfd0ec8 (127.16.217.3:35719)
I20260620 20:17:13.137413 17252 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260620 20:17:13.143579 27273 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:17:13.141995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57598 (local address 127.16.217.62:42933)
0620 20:17:13.142208 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0620 20:17:13.142213 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:17:13.142684 (+ 471us) server_negotiation.cc:407] Connection header received
0620 20:17:13.142877 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:17:13.142882 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:17:13.142944 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:17:13.143045 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:17:13.143387 (+ 342us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57598: BlockingRecv error: recv got EOF from 127.0.0.1:57598 (error 108)
Metrics: {"server-negotiator.queue_time_us":93}
I20260620 20:17:13.144497 17252 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu pid 27144
I20260620 20:17:13.148138 27541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:13.145368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44946 (local address 127.16.217.1:42605)
0620 20:17:13.145938 (+ 570us) server_negotiation.cc:206] Beginning negotiation
0620 20:17:13.145945 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:17:13.145972 (+ 27us) server_negotiation.cc:407] Connection header received
0620 20:17:13.146026 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:17:13.146033 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:17:13.146271 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:17:13.146404 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:17:13.147447 (+ 1043us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:17:13.147455 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:17:13.147467 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:17:13.147483 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:17:13.147536 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:17:13.147553 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:17:13.147719 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0620 20:17:13.147838 (+ 119us) server_negotiation.cc:299] Negotiation successful
0620 20:17:13.147921 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":372,"thread_start_us":229,"threads_started":1}
I20260620 20:17:13.149338 27266 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:17:13.235208 17252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu with pid 27144
I20260620 20:17:13.253522 17252 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu pid 27275
I20260620 20:17:13.257242 27543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:13.254630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54002 (local address 127.16.217.2:38939)
0620 20:17:13.255138 (+ 508us) server_negotiation.cc:206] Beginning negotiation
0620 20:17:13.255145 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:17:13.255178 (+ 33us) server_negotiation.cc:407] Connection header received
0620 20:17:13.255252 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:17:13.255259 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:17:13.255503 (+ 244us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:17:13.255692 (+ 189us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:17:13.256578 (+ 886us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:17:13.256585 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:17:13.256604 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:17:13.256617 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:17:13.256699 (+ 82us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:17:13.256717 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:17:13.256834 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0620 20:17:13.256907 (+ 73us) server_negotiation.cc:299] Negotiation successful
0620 20:17:13.256992 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":373,"thread_start_us":229,"threads_started":1}
I20260620 20:17:13.258805 27397 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:17:13.339344 17252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu with pid 27275
I20260620 20:17:13.355412 17252 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu pid 27405
I20260620 20:17:13.359511 27545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:13.356413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43278 (local address 127.16.217.3:35719)
0620 20:17:13.357079 (+ 666us) server_negotiation.cc:206] Beginning negotiation
0620 20:17:13.357087 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 20:17:13.357115 (+ 28us) server_negotiation.cc:407] Connection header received
0620 20:17:13.357185 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:17:13.357191 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:17:13.357407 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:17:13.357613 (+ 206us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:17:13.358790 (+ 1177us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:17:13.358796 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:17:13.358807 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:17:13.358823 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:17:13.358860 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:17:13.358873 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:17:13.359055 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0620 20:17:13.359129 (+ 74us) server_negotiation.cc:299] Negotiation successful
0620 20:17:13.359240 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":526,"thread_start_us":366,"threads_started":1}
I20260620 20:17:13.360725 27527 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:17:13.441489 17252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu with pid 27405
I20260620 20:17:13.457590 17252 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu pid 27054
I20260620 20:17:13.458881 27115 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:17:13.557803 17252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu with pid 27054
2026-06-20T20:17:13Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (2358 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986536687152-17252-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 20:17:13 dist-test-slave-2hpg krb5kdc[27550](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 20:17:13 dist-test-slave-2hpg krb5kdc[27550](info): set up 2 sockets
Jun 20 20:17:13 dist-test-slave-2hpg krb5kdc[27550](info): commencing operation
krb5kdc: starting...
W20260620 20:17:14.647096 17252 mini_kdc.cc:121] Time spent starting KDC: real 1.048s user 0.000s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986536687152-17252-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986536687152-17252-0/krb5kdc/test-user.keytab.
Jun 20 20:17:14 dist-test-slave-2hpg krb5kdc[27550](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986634, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T20:17:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T20:17:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.217.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.217.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.217.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.62.keytab.
Entry for principal kudu/127.16.217.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.62.keytab.
I20260620 20:17:14.875036 17252 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
/tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986536687152-17252-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986536687152-17252-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986536687152-17252-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986536687152-17252-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.217.62:36419
--webserver_interface=127.16.217.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.62.keytab
--principal=kudu/127.16.217.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.217.20:34353
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.217.62:36419
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986536687152-17252-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986536687152-17252-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986536687152-17252-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:17:15.215731 27563 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:17:15.216065 27563 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:17:15.216225 27563 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:17:15.223526 27563 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:17:15.223661 27563 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:17:15.223774 27563 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:17:15.223877 27563 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:17:15.223951 27563 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:17:15.233659 27563 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.217.20:34353
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986536687152-17252-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986536687152-17252-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.217.62:36419
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.62.keytab
--principal=kudu/127.16.217.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.217.62:36419
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986536687152-17252-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.217.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986536687152-17252-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:17:15.235141 27563 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:17:15.236466 27563 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:17:15.248265 27568 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:17:15.248265 27569 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:17:15.248303 27571 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:17:15.249101 27563 server_base.cc:1034] running on GCE node
I20260620 20:17:15.249795 27563 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:17:15.251165 27563 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:17:15.252380 27563 hybrid_clock.cc:648] HybridClock initialized: now 1781986635252326 us; error 38 us; skew 500 ppm
I20260620 20:17:15.252723 27563 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:17:15 dist-test-slave-2hpg krb5kdc[27550](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986635, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.217.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:17:15.255867 27563 init.cc:377] Logged in from keytab as kudu/127.16.217.62@KRBTEST.COM (short username kudu)
I20260620 20:17:15.257287 27563 webserver.cc:460] Webserver started at http://127.16.217.62:35105/ using document root <none> and password file <none>
I20260620 20:17:15.257951 27563 fs_manager.cc:359] Metadata directory not provided
I20260620 20:17:15.258054 27563 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:17:15.258296 27563 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:17:15.261116 27563 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986536687152-17252-0/minicluster-data/master-0/data/instance:
uuid: "a5463cecdcd042f18b8c6518b290da62"
format_stamp: "Formatted at 2026-06-20 20:17:15 on dist-test-slave-2hpg"
server_key: "54f483c5aa71528b4506f1283fc89be9"
server_key_iv: "8284f7d153a670fc2970ff980cf49471"
server_key_version: "encryptionkey@0"
I20260620 20:17:15.261849 27563 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986536687152-17252-0/minicluster-data/master-0/wal/instance:
uuid: "a5463cecdcd042f18b8c6518b290da62"
format_stamp: "Formatted at 2026-06-20 20:17:15 on dist-test-slave-2hpg"
server_key: "54f483c5aa71528b4506f1283fc89be9"
server_key_iv: "8284f7d153a670fc2970ff980cf49471"
server_key_version: "encryptionkey@0"
I20260620 20:17:15.267179 27563 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260620 20:17:15.270697 27578 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:17:15.272212 27563 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 20:17:15.272465 27563 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986536687152-17252-0/minicluster-data/master-0/data,/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986536687152-17252-0/minicluster-data/master-0/wal
uuid: "a5463cecdcd042f18b8c6518b290da62"
format_stamp: "Formatted at 2026-06-20 20:17:15 on dist-test-slave-2hpg"
server_key: "54f483c5aa71528b4506f1283fc89be9"
server_key_iv: "8284f7d153a670fc2970ff980cf49471"
server_key_version: "encryptionkey@0"
I20260620 20:17:15.272658 27563 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986536687152-17252-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986536687152-17252-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986536687152-17252-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:17:15.289387 27563 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:17:15.293463 27563 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:17:15.293752 27563 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:17:15.312260 27563 rpc_server.cc:307] RPC server started. Bound to: 127.16.217.62:36419
I20260620 20:17:15.312295 27629 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.217.62:36419 every 8 connection(s)
I20260620 20:17:15.313699 27563 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986536687152-17252-0/minicluster-data/master-0/data/info.pb
I20260620 20:17:15.316325 17252 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu as pid 27563
I20260620 20:17:15.316558 17252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986536687152-17252-0/minicluster-data/master-0/wal/instance
I20260620 20:17:15.317356 17252 external_mini_cluster.cc:1428] Setting key 7edea9ef805b78a16f2cdb0215e2b1c3
I20260620 20:17:15.318159 27630 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:17:15.332577 27630 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a5463cecdcd042f18b8c6518b290da62: Bootstrap starting.
Jun 20 20:17:15 dist-test-slave-2hpg krb5kdc[27550](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986634, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.217.62@KRBTEST.COM
I20260620 20:17:15.336594 27630 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a5463cecdcd042f18b8c6518b290da62: Neither blocks nor log segments found. Creating new log.
I20260620 20:17:15.337999 27630 log.cc:824] T 00000000000000000000000000000000 P a5463cecdcd042f18b8c6518b290da62: Log is configured to *not* fsync() on all Append() calls
I20260620 20:17:15.341468 27630 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a5463cecdcd042f18b8c6518b290da62: No bootstrap required, opened a new log
I20260620 20:17:15.342479 27633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:15.319432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43808 (local address 127.16.217.62:36419)
0620 20:17:15.320219 (+ 787us) server_negotiation.cc:206] Beginning negotiation
0620 20:17:15.320231 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 20:17:15.320276 (+ 45us) server_negotiation.cc:407] Connection header received
0620 20:17:15.322609 (+ 2333us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:17:15.322632 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:17:15.323068 (+ 436us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:17:15.323406 (+ 338us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:17:15.325215 (+ 1809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:15.326551 (+ 1336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:15.328323 (+ 1772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:15.329047 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:15.329194 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:17:15.333795 (+ 4601us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:17:15.333806 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:17:15.333825 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:17:15.333890 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:17:15.337802 (+ 3912us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:17:15.338556 (+ 754us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:17:15.338569 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:17:15.338586 (+ 17us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:17:15.338825 (+ 239us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:17:15.339287 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:17:15.339290 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:17:15.339292 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:17:15.339999 (+ 707us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:17:15.340294 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0620 20:17:15.340622 (+ 328us) server_negotiation.cc:299] Negotiation successful
0620 20:17:15.341052 (+ 430us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":592,"thread_start_us":389,"threads_started":1}
I20260620 20:17:15.349618 27630 raft_consensus.cc:348] T 00000000000000000000000000000000 P a5463cecdcd042f18b8c6518b290da62 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5463cecdcd042f18b8c6518b290da62" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 36419 } }
I20260620 20:17:15.350039 27630 raft_consensus.cc:374] T 00000000000000000000000000000000 P a5463cecdcd042f18b8c6518b290da62 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:17:15.350112 27630 raft_consensus.cc:729] T 00000000000000000000000000000000 P a5463cecdcd042f18b8c6518b290da62 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5463cecdcd042f18b8c6518b290da62, State: Initialized, Role: FOLLOWER
I20260620 20:17:15.350557 27630 consensus_queue.cc:260] T 00000000000000000000000000000000 P a5463cecdcd042f18b8c6518b290da62 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5463cecdcd042f18b8c6518b290da62" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 36419 } }
I20260620 20:17:15.350766 27630 raft_consensus.cc:388] T 00000000000000000000000000000000 P a5463cecdcd042f18b8c6518b290da62 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 20:17:15.350955 27630 raft_consensus.cc:482] T 00000000000000000000000000000000 P a5463cecdcd042f18b8c6518b290da62 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 20:17:15.351135 27630 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a5463cecdcd042f18b8c6518b290da62 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:17:15.353223 27630 raft_consensus.cc:504] T 00000000000000000000000000000000 P a5463cecdcd042f18b8c6518b290da62 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5463cecdcd042f18b8c6518b290da62" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 36419 } }
I20260620 20:17:15.353684 27630 leader_election.cc:302] T 00000000000000000000000000000000 P a5463cecdcd042f18b8c6518b290da62 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a5463cecdcd042f18b8c6518b290da62; no voters:
I20260620 20:17:15.354310 27630 leader_election.cc:288] T 00000000000000000000000000000000 P a5463cecdcd042f18b8c6518b290da62 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 20:17:15.354446 27635 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a5463cecdcd042f18b8c6518b290da62 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:17:15.355151 27635 raft_consensus.cc:686] T 00000000000000000000000000000000 P a5463cecdcd042f18b8c6518b290da62 [term 1 LEADER]: Becoming Leader. State: Replica: a5463cecdcd042f18b8c6518b290da62, State: Running, Role: LEADER
I20260620 20:17:15.355711 27635 consensus_queue.cc:237] T 00000000000000000000000000000000 P a5463cecdcd042f18b8c6518b290da62 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5463cecdcd042f18b8c6518b290da62" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 36419 } }
I20260620 20:17:15.356493 27630 sys_catalog.cc:564] T 00000000000000000000000000000000 P a5463cecdcd042f18b8c6518b290da62 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:17:15.358451 27636 sys_catalog.cc:455] T 00000000000000000000000000000000 P a5463cecdcd042f18b8c6518b290da62 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a5463cecdcd042f18b8c6518b290da62" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5463cecdcd042f18b8c6518b290da62" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 36419 } } }
I20260620 20:17:15.359045 27636 sys_catalog.cc:458] T 00000000000000000000000000000000 P a5463cecdcd042f18b8c6518b290da62 [sys.catalog]: This master's current role is: LEADER
I20260620 20:17:15.358955 27637 sys_catalog.cc:455] T 00000000000000000000000000000000 P a5463cecdcd042f18b8c6518b290da62 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a5463cecdcd042f18b8c6518b290da62. Latest consensus state: current_term: 1 leader_uuid: "a5463cecdcd042f18b8c6518b290da62" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5463cecdcd042f18b8c6518b290da62" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 36419 } } }
I20260620 20:17:15.359302 27637 sys_catalog.cc:458] T 00000000000000000000000000000000 P a5463cecdcd042f18b8c6518b290da62 [sys.catalog]: This master's current role is: LEADER
I20260620 20:17:15.361486 27643 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 20:17:15.366866 27643 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 20:17:15.374575 27643 catalog_manager.cc:1347] Generated new cluster ID: db3f98390f854fea9e57d11fd5cb1c94
I20260620 20:17:15.374693 27643 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 20:17:15.394384 27643 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 20:17:15.395815 27643 catalog_manager.cc:1504] Loading token signing keys...
I20260620 20:17:15.404244 27643 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a5463cecdcd042f18b8c6518b290da62: Generated new TSK 0
I20260620 20:17:15.405058 27643 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.217.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.217.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.217.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.1.keytab.
Entry for principal kudu/127.16.217.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.1.keytab.
I20260620 20:17:15.469190 17252 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
/tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986536687152-17252-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986536687152-17252-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986536687152-17252-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986536687152-17252-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.217.1:0
--local_ip_for_outbound_sockets=127.16.217.1
--webserver_interface=127.16.217.1
--webserver_port=0
--tserver_master_addrs=127.16.217.62:36419
--keytab_file=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.1.keytab
--principal=kudu/127.16.217.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.217.20:34353
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986536687152-17252-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986536687152-17252-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986536687152-17252-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:17:15.801700 27656 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:17:15.802023 27656 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:17:15.802120 27656 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:17:15.809453 27656 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:17:15.809574 27656 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:17:15.809692 27656 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.217.1
I20260620 20:17:15.819550 27656 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.217.20:34353
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986536687152-17252-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986536687152-17252-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.1.keytab
--principal=kudu/127.16.217.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.217.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986536687152-17252-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.217.1
--webserver_port=0
--tserver_master_addrs=127.16.217.62:36419
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.217.1
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986536687152-17252-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:17:15.820832 27656 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:17:15.822283 27656 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:17:15.835585 27662 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:17:15.835719 27661 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:17:15.835693 27664 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:17:15.836683 27656 server_base.cc:1034] running on GCE node
I20260620 20:17:15.837255 27656 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:17:15.838220 27656 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:17:15.839412 27656 hybrid_clock.cc:648] HybridClock initialized: now 1781986635839350 us; error 54 us; skew 500 ppm
I20260620 20:17:15.839751 27656 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:17:15 dist-test-slave-2hpg krb5kdc[27550](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986635, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.217.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:17:15.842774 27656 init.cc:377] Logged in from keytab as kudu/127.16.217.1@KRBTEST.COM (short username kudu)
I20260620 20:17:15.844338 27656 webserver.cc:460] Webserver started at http://127.16.217.1:39681/ using document root <none> and password file <none>
I20260620 20:17:15.845038 27656 fs_manager.cc:359] Metadata directory not provided
I20260620 20:17:15.845162 27656 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:17:15.845464 27656 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:17:15.848261 27656 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986536687152-17252-0/minicluster-data/ts-0/data/instance:
uuid: "90bd992782594257addca235ac0da375"
format_stamp: "Formatted at 2026-06-20 20:17:15 on dist-test-slave-2hpg"
server_key: "225c3be4affad198f8201e659dbeb19c"
server_key_iv: "0251042631c1eb94f169c9a7333826d9"
server_key_version: "encryptionkey@0"
I20260620 20:17:15.848982 27656 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986536687152-17252-0/minicluster-data/ts-0/wal/instance:
uuid: "90bd992782594257addca235ac0da375"
format_stamp: "Formatted at 2026-06-20 20:17:15 on dist-test-slave-2hpg"
server_key: "225c3be4affad198f8201e659dbeb19c"
server_key_iv: "0251042631c1eb94f169c9a7333826d9"
server_key_version: "encryptionkey@0"
I20260620 20:17:15.853997 27656 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260620 20:17:15.857456 27671 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:17:15.858916 27656 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 20:17:15.859077 27656 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986536687152-17252-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986536687152-17252-0/minicluster-data/ts-0/wal
uuid: "90bd992782594257addca235ac0da375"
format_stamp: "Formatted at 2026-06-20 20:17:15 on dist-test-slave-2hpg"
server_key: "225c3be4affad198f8201e659dbeb19c"
server_key_iv: "0251042631c1eb94f169c9a7333826d9"
server_key_version: "encryptionkey@0"
I20260620 20:17:15.859249 27656 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986536687152-17252-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986536687152-17252-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986536687152-17252-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:17:15.896206 27656 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:17:15.900262 27656 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:17:15.900571 27656 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:17:15.901569 27656 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:17:15.902926 27656 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:17:15.903015 27656 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:17:15.903136 27656 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:17:15.903205 27656 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:17:15.936388 27656 rpc_server.cc:307] RPC server started. Bound to: 127.16.217.1:45147
I20260620 20:17:15.936487 27783 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.217.1:45147 every 8 connection(s)
I20260620 20:17:15.938133 27656 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986536687152-17252-0/minicluster-data/ts-0/data/info.pb
I20260620 20:17:15.945823 17252 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu as pid 27656
I20260620 20:17:15.946061 17252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986536687152-17252-0/minicluster-data/ts-0/wal/instance
I20260620 20:17:15.946620 17252 external_mini_cluster.cc:1428] Setting key 087611ce85d0fbb2d20a344fb7949bb6
Jun 20 20:17:15 dist-test-slave-2hpg krb5kdc[27550](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986635, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.217.1@KRBTEST.COM for kudu/127.16.217.62@KRBTEST.COM
I20260620 20:17:15.959543 27786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:15.940634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.217.1:34109 (local address 127.16.217.62:36419)
0620 20:17:15.941051 (+ 417us) server_negotiation.cc:206] Beginning negotiation
0620 20:17:15.941074 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0620 20:17:15.942154 (+ 1080us) server_negotiation.cc:407] Connection header received
0620 20:17:15.943971 (+ 1817us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:17:15.943977 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:17:15.944068 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:17:15.944186 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:17:15.945603 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:15.946466 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:15.948467 (+ 2001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:15.949053 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:15.949185 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:17:15.952528 (+ 3343us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:17:15.952537 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:17:15.952545 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:17:15.952588 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:17:15.956076 (+ 3488us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:17:15.957082 (+ 1006us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:17:15.957086 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:17:15.957089 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:17:15.957199 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:17:15.957711 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:17:15.957714 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:17:15.957716 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:17:15.958138 (+ 422us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:17:15.958247 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0620 20:17:15.959133 (+ 886us) server_negotiation.cc:299] Negotiation successful
0620 20:17:15.959294 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":314,"thread_start_us":216,"threads_started":1}
I20260620 20:17:15.961199 27787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:15.941270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:36419 (local address 127.16.217.1:34109)
0620 20:17:15.942010 (+ 740us) negotiation.cc:107] Waiting for socket to connect
0620 20:17:15.942047 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 20:17:15.943704 (+ 1657us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:17:15.944409 (+ 705us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:17:15.944417 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:17:15.944903 (+ 486us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:17:15.945438 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:17:15.945453 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:17:15.946626 (+ 1173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:17:15.946631 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:17:15.948322 (+ 1691us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:17:15.948337 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:17:15.949279 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:17:15.949284 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:17:15.949484 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:17:15.950193 (+ 709us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:17:15.950214 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:17:15.952315 (+ 2101us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:17:15.956223 (+ 3908us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:17:15.956234 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:17:15.956253 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:17:15.956960 (+ 707us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:17:15.957328 (+ 368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:17:15.957336 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:17:15.957342 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:17:15.957584 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:17:15.958258 (+ 674us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:17:15.958269 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:17:15.958735 (+ 466us) client_negotiation.cc:769] Sending connection context
0620 20:17:15.959057 (+ 322us) client_negotiation.cc:240] Negotiation successful
0620 20:17:15.959701 (+ 644us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":459,"thread_start_us":265,"threads_started":1}
I20260620 20:17:15.962836 27784 heartbeater.cc:347] Connected to a master server at 127.16.217.62:36419
I20260620 20:17:15.963145 27784 heartbeater.cc:464] Registering TS with master...
I20260620 20:17:15.963915 27784 heartbeater.cc:511] Master 127.16.217.62:36419 requested a full tablet report, sending...
I20260620 20:17:15.966207 27595 ts_manager.cc:194] Registered new tserver with Master: 90bd992782594257addca235ac0da375 (127.16.217.1:45147)
I20260620 20:17:15.968451 27595 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.217.1@KRBTEST.COM'} at 127.16.217.1:34109
WARNING: no policy specified for kudu/127.16.217.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.217.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.217.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.2.keytab.
Entry for principal kudu/127.16.217.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.2.keytab.
I20260620 20:17:16.001029 17252 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
/tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986536687152-17252-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986536687152-17252-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986536687152-17252-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986536687152-17252-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.217.2:0
--local_ip_for_outbound_sockets=127.16.217.2
--webserver_interface=127.16.217.2
--webserver_port=0
--tserver_master_addrs=127.16.217.62:36419
--keytab_file=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.2.keytab
--principal=kudu/127.16.217.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.217.20:34353
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986536687152-17252-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986536687152-17252-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986536687152-17252-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:17:16.335476 27790 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:17:16.335788 27790 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:17:16.335888 27790 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:17:16.343297 27790 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:17:16.343417 27790 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:17:16.343539 27790 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.217.2
I20260620 20:17:16.353410 27790 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.217.20:34353
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986536687152-17252-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986536687152-17252-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.2.keytab
--principal=kudu/127.16.217.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.217.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986536687152-17252-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.217.2
--webserver_port=0
--tserver_master_addrs=127.16.217.62:36419
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.217.2
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986536687152-17252-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:17:16.354774 27790 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:17:16.356055 27790 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:17:16.369985 27795 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:17:16.370007 27796 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:17:16.370183 27798 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:17:16.375530 27790 server_base.cc:1034] running on GCE node
I20260620 20:17:16.376214 27790 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:17:16.377265 27790 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:17:16.378484 27790 hybrid_clock.cc:648] HybridClock initialized: now 1781986636378429 us; error 39 us; skew 500 ppm
I20260620 20:17:16.378860 27790 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:17:16 dist-test-slave-2hpg krb5kdc[27550](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986636, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.217.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:17:16.382380 27790 init.cc:377] Logged in from keytab as kudu/127.16.217.2@KRBTEST.COM (short username kudu)
I20260620 20:17:16.383978 27790 webserver.cc:460] Webserver started at http://127.16.217.2:40673/ using document root <none> and password file <none>
I20260620 20:17:16.384749 27790 fs_manager.cc:359] Metadata directory not provided
I20260620 20:17:16.384879 27790 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:17:16.385215 27790 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:17:16.388336 27790 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986536687152-17252-0/minicluster-data/ts-1/data/instance:
uuid: "8085d41a2b5941d0b20614771188c810"
format_stamp: "Formatted at 2026-06-20 20:17:16 on dist-test-slave-2hpg"
server_key: "96cc7d58b6204007bed481c5efd9350f"
server_key_iv: "810f50ee0226458f89b40c3d4edb1aee"
server_key_version: "encryptionkey@0"
I20260620 20:17:16.389148 27790 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986536687152-17252-0/minicluster-data/ts-1/wal/instance:
uuid: "8085d41a2b5941d0b20614771188c810"
format_stamp: "Formatted at 2026-06-20 20:17:16 on dist-test-slave-2hpg"
server_key: "96cc7d58b6204007bed481c5efd9350f"
server_key_iv: "810f50ee0226458f89b40c3d4edb1aee"
server_key_version: "encryptionkey@0"
I20260620 20:17:16.394524 27790 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260620 20:17:16.398144 27805 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:17:16.399595 27790 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 20:17:16.399785 27790 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986536687152-17252-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986536687152-17252-0/minicluster-data/ts-1/wal
uuid: "8085d41a2b5941d0b20614771188c810"
format_stamp: "Formatted at 2026-06-20 20:17:16 on dist-test-slave-2hpg"
server_key: "96cc7d58b6204007bed481c5efd9350f"
server_key_iv: "810f50ee0226458f89b40c3d4edb1aee"
server_key_version: "encryptionkey@0"
I20260620 20:17:16.399994 27790 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986536687152-17252-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986536687152-17252-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986536687152-17252-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:17:16.428440 27790 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:17:16.433259 27790 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:17:16.433620 27790 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:17:16.434768 27790 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:17:16.436313 27790 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:17:16.436437 27790 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:17:16.436619 27790 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:17:16.436700 27790 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:17:16.473737 27790 rpc_server.cc:307] RPC server started. Bound to: 127.16.217.2:35297
I20260620 20:17:16.473783 27917 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.217.2:35297 every 8 connection(s)
I20260620 20:17:16.475142 27790 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986536687152-17252-0/minicluster-data/ts-1/data/info.pb
I20260620 20:17:16.483999 17252 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu as pid 27790
I20260620 20:17:16.484208 17252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986536687152-17252-0/minicluster-data/ts-1/wal/instance
I20260620 20:17:16.484699 17252 external_mini_cluster.cc:1428] Setting key bce657729c0a6a2d94feabefc5f31f25
Jun 20 20:17:16 dist-test-slave-2hpg krb5kdc[27550](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986636, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.217.2@KRBTEST.COM for kudu/127.16.217.62@KRBTEST.COM
I20260620 20:17:16.498262 27920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:16.478047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.217.2:50369 (local address 127.16.217.62:36419)
0620 20:17:16.478536 (+ 489us) server_negotiation.cc:206] Beginning negotiation
0620 20:17:16.478543 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:17:16.479652 (+ 1109us) server_negotiation.cc:407] Connection header received
0620 20:17:16.481410 (+ 1758us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:17:16.481415 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:17:16.481481 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:17:16.481583 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:17:16.483084 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:16.484373 (+ 1289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:16.486320 (+ 1947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:16.486934 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:16.487059 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:17:16.490576 (+ 3517us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:17:16.490585 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:17:16.490592 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:17:16.490636 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:17:16.494665 (+ 4029us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:17:16.495656 (+ 991us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:17:16.495660 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:17:16.495663 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:17:16.495832 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:17:16.496352 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:17:16.496355 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:17:16.496358 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:17:16.496626 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:17:16.496783 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0620 20:17:16.497650 (+ 867us) server_negotiation.cc:299] Negotiation successful
0620 20:17:16.497937 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":355,"thread_start_us":242,"threads_started":1}
I20260620 20:17:16.499696 27921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:16.478801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:36419 (local address 127.16.217.2:50369)
0620 20:17:16.479486 (+ 685us) negotiation.cc:107] Waiting for socket to connect
0620 20:17:16.479529 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 20:17:16.481120 (+ 1591us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:17:16.481783 (+ 663us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:17:16.481791 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:17:16.482368 (+ 577us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:17:16.482919 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:17:16.482935 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:17:16.484519 (+ 1584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:17:16.484523 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:17:16.486193 (+ 1670us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:17:16.486206 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:17:16.487134 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:17:16.487139 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:17:16.487340 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:17:16.488067 (+ 727us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:17:16.488089 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:17:16.490330 (+ 2241us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:17:16.494825 (+ 4495us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:17:16.494837 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:17:16.494857 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:17:16.495522 (+ 665us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:17:16.495949 (+ 427us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:17:16.495957 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:17:16.495963 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:17:16.496228 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:17:16.496787 (+ 559us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:17:16.496797 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:17:16.497263 (+ 466us) client_negotiation.cc:769] Sending connection context
0620 20:17:16.497558 (+ 295us) client_negotiation.cc:240] Negotiation successful
0620 20:17:16.498182 (+ 624us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":465,"thread_start_us":283,"threads_started":1}
I20260620 20:17:16.501233 27918 heartbeater.cc:347] Connected to a master server at 127.16.217.62:36419
I20260620 20:17:16.501573 27918 heartbeater.cc:464] Registering TS with master...
I20260620 20:17:16.502418 27918 heartbeater.cc:511] Master 127.16.217.62:36419 requested a full tablet report, sending...
I20260620 20:17:16.503907 27595 ts_manager.cc:194] Registered new tserver with Master: 8085d41a2b5941d0b20614771188c810 (127.16.217.2:35297)
I20260620 20:17:16.504964 27595 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.217.2@KRBTEST.COM'} at 127.16.217.2:50369
WARNING: no policy specified for kudu/127.16.217.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.217.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.217.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.3.keytab.
Entry for principal kudu/127.16.217.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.3.keytab.
I20260620 20:17:16.537149 17252 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
/tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986536687152-17252-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986536687152-17252-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986536687152-17252-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986536687152-17252-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.217.3:0
--local_ip_for_outbound_sockets=127.16.217.3
--webserver_interface=127.16.217.3
--webserver_port=0
--tserver_master_addrs=127.16.217.62:36419
--keytab_file=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.3.keytab
--principal=kudu/127.16.217.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.217.20:34353
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986536687152-17252-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986536687152-17252-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986536687152-17252-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:17:16.863037 27924 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:17:16.863360 27924 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:17:16.863533 27924 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:17:16.870986 27924 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:17:16.871133 27924 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:17:16.871325 27924 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.217.3
I20260620 20:17:16.881119 27924 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.217.20:34353
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986536687152-17252-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986536687152-17252-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.3.keytab
--principal=kudu/127.16.217.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.217.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986536687152-17252-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.217.3
--webserver_port=0
--tserver_master_addrs=127.16.217.62:36419
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.217.3
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986536687152-17252-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:17:16.882643 27924 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:17:16.883926 27924 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:17:16.896689 27929 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:17:16.896715 27930 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:17:16.896767 27932 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:17:16.897927 27924 server_base.cc:1034] running on GCE node
I20260620 20:17:16.898569 27924 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:17:16.899528 27924 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:17:16.900760 27924 hybrid_clock.cc:648] HybridClock initialized: now 1781986636900694 us; error 66 us; skew 500 ppm
I20260620 20:17:16.901080 27924 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:17:16 dist-test-slave-2hpg krb5kdc[27550](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986636, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.217.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:17:16.904340 27924 init.cc:377] Logged in from keytab as kudu/127.16.217.3@KRBTEST.COM (short username kudu)
I20260620 20:17:16.905855 27924 webserver.cc:460] Webserver started at http://127.16.217.3:45107/ using document root <none> and password file <none>
I20260620 20:17:16.906599 27924 fs_manager.cc:359] Metadata directory not provided
I20260620 20:17:16.906759 27924 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:17:16.907079 27924 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:17:16.910033 27924 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986536687152-17252-0/minicluster-data/ts-2/data/instance:
uuid: "2772c95187a54a99901b430205a53856"
format_stamp: "Formatted at 2026-06-20 20:17:16 on dist-test-slave-2hpg"
server_key: "40e70e37206bd24f80ce2836acf32712"
server_key_iv: "81fc0b6fc0547b3bbcf7feefe9c55993"
server_key_version: "encryptionkey@0"
I20260620 20:17:16.910781 27924 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986536687152-17252-0/minicluster-data/ts-2/wal/instance:
uuid: "2772c95187a54a99901b430205a53856"
format_stamp: "Formatted at 2026-06-20 20:17:16 on dist-test-slave-2hpg"
server_key: "40e70e37206bd24f80ce2836acf32712"
server_key_iv: "81fc0b6fc0547b3bbcf7feefe9c55993"
server_key_version: "encryptionkey@0"
I20260620 20:17:16.916199 27924 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 20:17:16.919894 27939 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:17:16.921317 27924 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 20:17:16.921506 27924 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986536687152-17252-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986536687152-17252-0/minicluster-data/ts-2/wal
uuid: "2772c95187a54a99901b430205a53856"
format_stamp: "Formatted at 2026-06-20 20:17:16 on dist-test-slave-2hpg"
server_key: "40e70e37206bd24f80ce2836acf32712"
server_key_iv: "81fc0b6fc0547b3bbcf7feefe9c55993"
server_key_version: "encryptionkey@0"
I20260620 20:17:16.921708 27924 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986536687152-17252-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986536687152-17252-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986536687152-17252-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:17:16.939538 27924 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:17:16.944068 27924 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:17:16.944307 27924 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:17:16.945194 27924 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:17:16.946518 27924 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:17:16.946589 27924 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:17:16.946672 27924 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:17:16.946717 27924 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:17:16.971521 27784 heartbeater.cc:503] Master 127.16.217.62:36419 was elected leader, sending a full tablet report...
I20260620 20:17:16.981223 27924 rpc_server.cc:307] RPC server started. Bound to: 127.16.217.3:42887
I20260620 20:17:16.981312 28051 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.217.3:42887 every 8 connection(s)
I20260620 20:17:16.982586 27924 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986536687152-17252-0/minicluster-data/ts-2/data/info.pb
I20260620 20:17:16.989979 17252 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu as pid 27924
I20260620 20:17:16.990190 17252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986536687152-17252-0/minicluster-data/ts-2/wal/instance
I20260620 20:17:16.990711 17252 external_mini_cluster.cc:1428] Setting key 6acd241d0a41f865aae4021c86d90d38
Jun 20 20:17:16 dist-test-slave-2hpg krb5kdc[27550](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986636, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.217.3@KRBTEST.COM for kudu/127.16.217.62@KRBTEST.COM
I20260620 20:17:17.002895 27920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:16.985260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.217.3:58749 (local address 127.16.217.62:36419)
0620 20:17:16.985466 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0620 20:17:16.985473 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:17:16.986847 (+ 1374us) server_negotiation.cc:407] Connection header received
0620 20:17:16.988561 (+ 1714us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:17:16.988565 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:17:16.988645 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:17:16.988733 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:17:16.990515 (+ 1782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:16.991359 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:16.993456 (+ 2097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:16.993979 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:16.994079 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:17:16.997480 (+ 3401us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:17:16.997485 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:17:16.997488 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:17:16.997520 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:17:17.000027 (+ 2507us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:17:17.000866 (+ 839us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:17:17.000870 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:17:17.000872 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:17:17.000977 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:17:17.001380 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:17:17.001383 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:17:17.001385 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:17:17.001635 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:17:17.001760 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0620 20:17:17.002487 (+ 727us) server_negotiation.cc:299] Negotiation successful
0620 20:17:17.002658 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260620 20:17:17.004194 28054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:16.985789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:36419 (local address 127.16.217.3:58749)
0620 20:17:16.986673 (+ 884us) negotiation.cc:107] Waiting for socket to connect
0620 20:17:16.986719 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 20:17:16.988318 (+ 1599us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:17:16.988949 (+ 631us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:17:16.988956 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:17:16.989463 (+ 507us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:17:16.990345 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:17:16.990372 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:17:16.991518 (+ 1146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:17:16.991523 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:17:16.993292 (+ 1769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:17:16.993304 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:17:16.994271 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:17:16.994276 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:17:16.994476 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:17:16.995093 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:17:16.995116 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:17:16.997293 (+ 2177us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:17:17.000160 (+ 2867us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:17:17.000168 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:17:17.000184 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:17:17.000751 (+ 567us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:17:17.001070 (+ 319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:17:17.001073 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:17:17.001075 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:17:17.001277 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:17:17.001772 (+ 495us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:17:17.001778 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:17:17.002167 (+ 389us) client_negotiation.cc:769] Sending connection context
0620 20:17:17.002407 (+ 240us) client_negotiation.cc:240] Negotiation successful
0620 20:17:17.002847 (+ 440us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":644,"thread_start_us":386,"threads_started":1}
I20260620 20:17:17.005703 28052 heartbeater.cc:347] Connected to a master server at 127.16.217.62:36419
I20260620 20:17:17.006076 28052 heartbeater.cc:464] Registering TS with master...
I20260620 20:17:17.006908 28052 heartbeater.cc:511] Master 127.16.217.62:36419 requested a full tablet report, sending...
I20260620 20:17:17.008281 27595 ts_manager.cc:194] Registered new tserver with Master: 2772c95187a54a99901b430205a53856 (127.16.217.3:42887)
I20260620 20:17:17.009106 27595 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.217.3@KRBTEST.COM'} at 127.16.217.3:58749
I20260620 20:17:17.018739 17252 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 20:17:17.036027 27920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:17.023261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43820 (local address 127.16.217.62:36419)
0620 20:17:17.023425 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0620 20:17:17.023430 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:17:17.024133 (+ 703us) server_negotiation.cc:407] Connection header received
0620 20:17:17.024289 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:17:17.024292 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:17:17.024353 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:17:17.024437 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:17:17.026085 (+ 1648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:17.026909 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:17.028154 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:17.028535 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:17.028632 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:17:17.031202 (+ 2570us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:17:17.031206 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:17:17.031210 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:17:17.031241 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:17:17.033834 (+ 2593us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:17:17.034414 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:17:17.034418 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:17:17.034420 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:17:17.034540 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:17:17.034850 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:17:17.034853 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:17:17.034855 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:17:17.035136 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:17:17.035326 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0620 20:17:17.035598 (+ 272us) server_negotiation.cc:299] Negotiation successful
0620 20:17:17.035768 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 20:17:17.040129 27595 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43820:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 20:17:17.046900 27595 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 20:17:17.078691 28064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:17.058901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.1:45147 (local address 127.0.0.1:58550)
0620 20:17:17.067040 (+ 8139us) negotiation.cc:107] Waiting for socket to connect
0620 20:17:17.067101 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0620 20:17:17.067349 (+ 248us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:17:17.068756 (+ 1407us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:17:17.068767 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:17:17.068808 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:17:17.069353 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:17:17.069370 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:17:17.071988 (+ 2618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:17:17.071996 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:17:17.074872 (+ 2876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:17:17.074892 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:17:17.077368 (+ 2476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:17:17.077377 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:17:17.077766 (+ 389us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:17:17.077831 (+ 65us) client_negotiation.cc:769] Sending connection context
0620 20:17:17.078001 (+ 170us) client_negotiation.cc:240] Negotiation successful
0620 20:17:17.078188 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7809,"mutex_wait_us":3,"thread_start_us":7636,"threads_started":1}
I20260620 20:17:17.079265 28066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:17.059216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58550 (local address 127.16.217.1:45147)
0620 20:17:17.066532 (+ 7316us) server_negotiation.cc:206] Beginning negotiation
0620 20:17:17.066544 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 20:17:17.067694 (+ 1150us) server_negotiation.cc:407] Connection header received
0620 20:17:17.067843 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:17:17.067854 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:17:17.068186 (+ 332us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:17:17.068412 (+ 226us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:17:17.069597 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:17.071775 (+ 2178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:17.075104 (+ 3329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:17.076757 (+ 1653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:17.076923 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:17:17.077224 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0620 20:17:17.078410 (+ 1186us) server_negotiation.cc:299] Negotiation successful
0620 20:17:17.078580 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7090,"thread_start_us":6895,"threads_started":1}
I20260620 20:17:17.082557 28067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:17.059646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38598 (local address 127.16.217.3:42887)
0620 20:17:17.071646 (+ 12000us) server_negotiation.cc:206] Beginning negotiation
0620 20:17:17.071659 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 20:17:17.072817 (+ 1158us) server_negotiation.cc:407] Connection header received
0620 20:17:17.072940 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:17:17.072951 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:17:17.073266 (+ 315us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:17:17.073480 (+ 214us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:17:17.074638 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:17.076746 (+ 2108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:17.080209 (+ 3463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:17.081153 (+ 944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:17.081229 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:17:17.081395 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0620 20:17:17.082066 (+ 671us) server_negotiation.cc:299] Negotiation successful
0620 20:17:17.082223 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11740,"thread_start_us":12002,"threads_started":1}
I20260620 20:17:17.082595 28065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:17.059263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.3:42887 (local address 127.0.0.1:38598)
0620 20:17:17.072443 (+ 13180us) negotiation.cc:107] Waiting for socket to connect
0620 20:17:17.072471 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0620 20:17:17.072645 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:17:17.073795 (+ 1150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:17:17.073803 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:17:17.073835 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:17:17.074403 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:17:17.074418 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:17:17.076930 (+ 2512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:17:17.076937 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:17:17.080021 (+ 3084us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:17:17.080034 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:17:17.081527 (+ 1493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:17:17.081531 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:17:17.081797 (+ 266us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:17:17.081846 (+ 49us) client_negotiation.cc:769] Sending connection context
0620 20:17:17.081985 (+ 139us) client_negotiation.cc:240] Negotiation successful
0620 20:17:17.082206 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13006,"mutex_wait_us":674,"thread_start_us":12789,"threads_started":1}
I20260620 20:17:17.083323 27719 tablet_service.cc:1467] Processing CreateTablet for tablet a349104b859649a0b49aea41af53ff3e (DEFAULT_TABLE table=test-table [id=9c1b6098b6b84ad28870fcb04ae3c138]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:17:17.084786 27719 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a349104b859649a0b49aea41af53ff3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:17:17.086328 28069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:17.073350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.2:35297 (local address 127.0.0.1:40040)
0620 20:17:17.078058 (+ 4708us) negotiation.cc:107] Waiting for socket to connect
0620 20:17:17.078072 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 20:17:17.078216 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:17:17.079626 (+ 1410us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:17:17.079631 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:17:17.079656 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:17:17.079907 (+ 251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:17:17.079914 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:17:17.081636 (+ 1722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:17:17.081640 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:17:17.084024 (+ 2384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:17:17.084035 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:17:17.085542 (+ 1507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:17:17.085548 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:17:17.085713 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:17:17.085745 (+ 32us) client_negotiation.cc:769] Sending connection context
0620 20:17:17.085855 (+ 110us) client_negotiation.cc:240] Negotiation successful
0620 20:17:17.085993 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4591,"thread_start_us":4733,"threads_started":1}
I20260620 20:17:17.086385 28068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:17.067837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40040 (local address 127.16.217.2:35297)
0620 20:17:17.078065 (+ 10228us) server_negotiation.cc:206] Beginning negotiation
0620 20:17:17.078073 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 20:17:17.078175 (+ 102us) server_negotiation.cc:407] Connection header received
0620 20:17:17.078400 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:17:17.078409 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:17:17.079184 (+ 775us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:17:17.079364 (+ 180us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:17:17.080068 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:17.081488 (+ 1420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:17.084161 (+ 2673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:17.085345 (+ 1184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:17.085501 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:17:17.085690 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0620 20:17:17.085925 (+ 235us) server_negotiation.cc:299] Negotiation successful
0620 20:17:17.086082 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10071,"thread_start_us":9918,"threads_started":1}
I20260620 20:17:17.086303 27987 tablet_service.cc:1467] Processing CreateTablet for tablet a349104b859649a0b49aea41af53ff3e (DEFAULT_TABLE table=test-table [id=9c1b6098b6b84ad28870fcb04ae3c138]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:17:17.087705 27987 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a349104b859649a0b49aea41af53ff3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:17:17.089619 27853 tablet_service.cc:1467] Processing CreateTablet for tablet a349104b859649a0b49aea41af53ff3e (DEFAULT_TABLE table=test-table [id=9c1b6098b6b84ad28870fcb04ae3c138]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:17:17.090862 27853 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a349104b859649a0b49aea41af53ff3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:17:17.099370 28070 tablet_bootstrap.cc:492] T a349104b859649a0b49aea41af53ff3e P 90bd992782594257addca235ac0da375: Bootstrap starting.
I20260620 20:17:17.102005 28071 tablet_bootstrap.cc:492] T a349104b859649a0b49aea41af53ff3e P 2772c95187a54a99901b430205a53856: Bootstrap starting.
I20260620 20:17:17.103803 28072 tablet_bootstrap.cc:492] T a349104b859649a0b49aea41af53ff3e P 8085d41a2b5941d0b20614771188c810: Bootstrap starting.
I20260620 20:17:17.104650 28070 tablet_bootstrap.cc:654] T a349104b859649a0b49aea41af53ff3e P 90bd992782594257addca235ac0da375: Neither blocks nor log segments found. Creating new log.
I20260620 20:17:17.105527 28071 tablet_bootstrap.cc:654] T a349104b859649a0b49aea41af53ff3e P 2772c95187a54a99901b430205a53856: Neither blocks nor log segments found. Creating new log.
I20260620 20:17:17.105928 28070 log.cc:824] T a349104b859649a0b49aea41af53ff3e P 90bd992782594257addca235ac0da375: Log is configured to *not* fsync() on all Append() calls
I20260620 20:17:17.106801 28071 log.cc:824] T a349104b859649a0b49aea41af53ff3e P 2772c95187a54a99901b430205a53856: Log is configured to *not* fsync() on all Append() calls
I20260620 20:17:17.107378 28072 tablet_bootstrap.cc:654] T a349104b859649a0b49aea41af53ff3e P 8085d41a2b5941d0b20614771188c810: Neither blocks nor log segments found. Creating new log.
I20260620 20:17:17.108397 28072 log.cc:824] T a349104b859649a0b49aea41af53ff3e P 8085d41a2b5941d0b20614771188c810: Log is configured to *not* fsync() on all Append() calls
I20260620 20:17:17.109071 28070 tablet_bootstrap.cc:492] T a349104b859649a0b49aea41af53ff3e P 90bd992782594257addca235ac0da375: No bootstrap required, opened a new log
I20260620 20:17:17.109361 28070 ts_tablet_manager.cc:1397] T a349104b859649a0b49aea41af53ff3e P 90bd992782594257addca235ac0da375: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.003s
I20260620 20:17:17.109772 28071 tablet_bootstrap.cc:492] T a349104b859649a0b49aea41af53ff3e P 2772c95187a54a99901b430205a53856: No bootstrap required, opened a new log
I20260620 20:17:17.110062 28071 ts_tablet_manager.cc:1397] T a349104b859649a0b49aea41af53ff3e P 2772c95187a54a99901b430205a53856: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.003s
I20260620 20:17:17.111423 28072 tablet_bootstrap.cc:492] T a349104b859649a0b49aea41af53ff3e P 8085d41a2b5941d0b20614771188c810: No bootstrap required, opened a new log
I20260620 20:17:17.111667 28072 ts_tablet_manager.cc:1397] T a349104b859649a0b49aea41af53ff3e P 8085d41a2b5941d0b20614771188c810: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260620 20:17:17.117316 28070 raft_consensus.cc:348] T a349104b859649a0b49aea41af53ff3e P 90bd992782594257addca235ac0da375 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90bd992782594257addca235ac0da375" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 45147 } } peers { permanent_uuid: "2772c95187a54a99901b430205a53856" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42887 } } peers { permanent_uuid: "8085d41a2b5941d0b20614771188c810" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35297 } }
I20260620 20:17:17.117677 28070 raft_consensus.cc:374] T a349104b859649a0b49aea41af53ff3e P 90bd992782594257addca235ac0da375 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:17:17.117821 28070 raft_consensus.cc:729] T a349104b859649a0b49aea41af53ff3e P 90bd992782594257addca235ac0da375 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90bd992782594257addca235ac0da375, State: Initialized, Role: FOLLOWER
I20260620 20:17:17.118471 28070 consensus_queue.cc:260] T a349104b859649a0b49aea41af53ff3e P 90bd992782594257addca235ac0da375 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90bd992782594257addca235ac0da375" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 45147 } } peers { permanent_uuid: "2772c95187a54a99901b430205a53856" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42887 } } peers { permanent_uuid: "8085d41a2b5941d0b20614771188c810" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35297 } }
I20260620 20:17:17.120239 28070 ts_tablet_manager.cc:1428] T a349104b859649a0b49aea41af53ff3e P 90bd992782594257addca235ac0da375: Time spent starting tablet: real 0.011s user 0.010s sys 0.002s
I20260620 20:17:17.120654 28071 raft_consensus.cc:348] T a349104b859649a0b49aea41af53ff3e P 2772c95187a54a99901b430205a53856 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90bd992782594257addca235ac0da375" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 45147 } } peers { permanent_uuid: "2772c95187a54a99901b430205a53856" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42887 } } peers { permanent_uuid: "8085d41a2b5941d0b20614771188c810" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35297 } }
I20260620 20:17:17.121066 28071 raft_consensus.cc:374] T a349104b859649a0b49aea41af53ff3e P 2772c95187a54a99901b430205a53856 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:17:17.121192 28071 raft_consensus.cc:729] T a349104b859649a0b49aea41af53ff3e P 2772c95187a54a99901b430205a53856 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2772c95187a54a99901b430205a53856, State: Initialized, Role: FOLLOWER
I20260620 20:17:17.121955 28071 consensus_queue.cc:260] T a349104b859649a0b49aea41af53ff3e P 2772c95187a54a99901b430205a53856 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90bd992782594257addca235ac0da375" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 45147 } } peers { permanent_uuid: "2772c95187a54a99901b430205a53856" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42887 } } peers { permanent_uuid: "8085d41a2b5941d0b20614771188c810" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35297 } }
I20260620 20:17:17.122159 28072 raft_consensus.cc:348] T a349104b859649a0b49aea41af53ff3e P 8085d41a2b5941d0b20614771188c810 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90bd992782594257addca235ac0da375" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 45147 } } peers { permanent_uuid: "2772c95187a54a99901b430205a53856" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42887 } } peers { permanent_uuid: "8085d41a2b5941d0b20614771188c810" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35297 } }
I20260620 20:17:17.122617 28072 raft_consensus.cc:374] T a349104b859649a0b49aea41af53ff3e P 8085d41a2b5941d0b20614771188c810 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:17:17.122750 28072 raft_consensus.cc:729] T a349104b859649a0b49aea41af53ff3e P 8085d41a2b5941d0b20614771188c810 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8085d41a2b5941d0b20614771188c810, State: Initialized, Role: FOLLOWER
I20260620 20:17:17.123301 28072 consensus_queue.cc:260] T a349104b859649a0b49aea41af53ff3e P 8085d41a2b5941d0b20614771188c810 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90bd992782594257addca235ac0da375" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 45147 } } peers { permanent_uuid: "2772c95187a54a99901b430205a53856" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42887 } } peers { permanent_uuid: "8085d41a2b5941d0b20614771188c810" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35297 } }
I20260620 20:17:17.123831 28052 heartbeater.cc:503] Master 127.16.217.62:36419 was elected leader, sending a full tablet report...
I20260620 20:17:17.123878 28071 ts_tablet_manager.cc:1428] T a349104b859649a0b49aea41af53ff3e P 2772c95187a54a99901b430205a53856: Time spent starting tablet: real 0.014s user 0.012s sys 0.000s
I20260620 20:17:17.125396 27918 heartbeater.cc:503] Master 127.16.217.62:36419 was elected leader, sending a full tablet report...
I20260620 20:17:17.126124 28072 ts_tablet_manager.cc:1428] T a349104b859649a0b49aea41af53ff3e P 8085d41a2b5941d0b20614771188c810: Time spent starting tablet: real 0.014s user 0.011s sys 0.005s
I20260620 20:17:17.141604 28078 raft_consensus.cc:482] T a349104b859649a0b49aea41af53ff3e P 8085d41a2b5941d0b20614771188c810 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 20:17:17.141949 28078 raft_consensus.cc:504] T a349104b859649a0b49aea41af53ff3e P 8085d41a2b5941d0b20614771188c810 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90bd992782594257addca235ac0da375" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 45147 } } peers { permanent_uuid: "2772c95187a54a99901b430205a53856" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42887 } } peers { permanent_uuid: "8085d41a2b5941d0b20614771188c810" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35297 } }
I20260620 20:17:17.143611 28078 leader_election.cc:288] T a349104b859649a0b49aea41af53ff3e P 8085d41a2b5941d0b20614771188c810 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 90bd992782594257addca235ac0da375 (127.16.217.1:45147), 2772c95187a54a99901b430205a53856 (127.16.217.3:42887)
I20260620 20:17:17.152626 28067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:17.144251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.217.2:51719 (local address 127.16.217.3:42887)
0620 20:17:17.144489 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0620 20:17:17.144498 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 20:17:17.144884 (+ 386us) server_negotiation.cc:407] Connection header received
0620 20:17:17.145138 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:17:17.145146 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:17:17.145227 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:17:17.145332 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:17:17.146019 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:17.147155 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:17.150206 (+ 3051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:17.151425 (+ 1219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:17.151587 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:17:17.151660 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 20:17:17.152242 (+ 582us) server_negotiation.cc:299] Negotiation successful
0620 20:17:17.152371 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260620 20:17:17.152630 28066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:17.143931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.217.2:46449 (local address 127.16.217.1:45147)
0620 20:17:17.144213 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0620 20:17:17.144223 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 20:17:17.145158 (+ 935us) server_negotiation.cc:407] Connection header received
0620 20:17:17.145318 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:17:17.145322 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:17:17.145391 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:17:17.145479 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:17:17.146224 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:17.147250 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:17.150224 (+ 2974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:17.151423 (+ 1199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:17.151587 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:17:17.151660 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 20:17:17.152242 (+ 582us) server_negotiation.cc:299] Negotiation successful
0620 20:17:17.152371 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260620 20:17:17.152879 28080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:17.143981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.1:45147 (local address 127.16.217.2:46449)
0620 20:17:17.145065 (+ 1084us) negotiation.cc:107] Waiting for socket to connect
0620 20:17:17.145080 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 20:17:17.145177 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:17:17.145511 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:17:17.145514 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:17:17.145535 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:17:17.146075 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:17:17.146085 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:17:17.147374 (+ 1289us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:17:17.147382 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:17:17.150038 (+ 2656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:17:17.150052 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:17:17.151736 (+ 1684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:17:17.151743 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:17:17.151998 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:17:17.152086 (+ 88us) client_negotiation.cc:769] Sending connection context
0620 20:17:17.152192 (+ 106us) client_negotiation.cc:240] Negotiation successful
0620 20:17:17.152374 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":979,"mutex_wait_us":269,"thread_start_us":780,"threads_started":1}
I20260620 20:17:17.153084 28079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:17.143881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.3:42887 (local address 127.16.217.2:51719)
0620 20:17:17.144763 (+ 882us) negotiation.cc:107] Waiting for socket to connect
0620 20:17:17.144783 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 20:17:17.144970 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:17:17.145350 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:17:17.145352 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:17:17.145369 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:17:17.145847 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:17:17.145883 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:17:17.147341 (+ 1458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:17:17.147349 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:17:17.150044 (+ 2695us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:17:17.150052 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:17:17.151740 (+ 1688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:17:17.151744 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:17:17.152003 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:17:17.152086 (+ 83us) client_negotiation.cc:769] Sending connection context
0620 20:17:17.152199 (+ 113us) client_negotiation.cc:240] Negotiation successful
0620 20:17:17.152374 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":749,"mutex_wait_us":320,"thread_start_us":545,"threads_started":1}
I20260620 20:17:17.153859 27739 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a349104b859649a0b49aea41af53ff3e" candidate_uuid: "8085d41a2b5941d0b20614771188c810" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90bd992782594257addca235ac0da375" is_pre_election: true
I20260620 20:17:17.154119 28007 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a349104b859649a0b49aea41af53ff3e" candidate_uuid: "8085d41a2b5941d0b20614771188c810" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2772c95187a54a99901b430205a53856" is_pre_election: true
I20260620 20:17:17.154546 27739 raft_consensus.cc:2454] T a349104b859649a0b49aea41af53ff3e P 90bd992782594257addca235ac0da375 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8085d41a2b5941d0b20614771188c810 in term 0.
I20260620 20:17:17.154639 28007 raft_consensus.cc:2454] T a349104b859649a0b49aea41af53ff3e P 2772c95187a54a99901b430205a53856 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8085d41a2b5941d0b20614771188c810 in term 0.
I20260620 20:17:17.155230 27806 leader_election.cc:302] T a349104b859649a0b49aea41af53ff3e P 8085d41a2b5941d0b20614771188c810 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8085d41a2b5941d0b20614771188c810, 90bd992782594257addca235ac0da375; no voters:
I20260620 20:17:17.155623 28078 raft_consensus.cc:2781] T a349104b859649a0b49aea41af53ff3e P 8085d41a2b5941d0b20614771188c810 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 20:17:17.155776 28078 raft_consensus.cc:482] T a349104b859649a0b49aea41af53ff3e P 8085d41a2b5941d0b20614771188c810 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 20:17:17.155887 28078 raft_consensus.cc:3037] T a349104b859649a0b49aea41af53ff3e P 8085d41a2b5941d0b20614771188c810 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:17:17.158239 28078 raft_consensus.cc:504] T a349104b859649a0b49aea41af53ff3e P 8085d41a2b5941d0b20614771188c810 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90bd992782594257addca235ac0da375" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 45147 } } peers { permanent_uuid: "2772c95187a54a99901b430205a53856" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42887 } } peers { permanent_uuid: "8085d41a2b5941d0b20614771188c810" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35297 } }
I20260620 20:17:17.158866 28078 leader_election.cc:288] T a349104b859649a0b49aea41af53ff3e P 8085d41a2b5941d0b20614771188c810 [CANDIDATE]: Term 1 election: Requested vote from peers 90bd992782594257addca235ac0da375 (127.16.217.1:45147), 2772c95187a54a99901b430205a53856 (127.16.217.3:42887)
I20260620 20:17:17.159286 27739 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a349104b859649a0b49aea41af53ff3e" candidate_uuid: "8085d41a2b5941d0b20614771188c810" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90bd992782594257addca235ac0da375"
I20260620 20:17:17.159497 27739 raft_consensus.cc:3037] T a349104b859649a0b49aea41af53ff3e P 90bd992782594257addca235ac0da375 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:17:17.159428 28007 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a349104b859649a0b49aea41af53ff3e" candidate_uuid: "8085d41a2b5941d0b20614771188c810" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2772c95187a54a99901b430205a53856"
I20260620 20:17:17.159679 28007 raft_consensus.cc:3037] T a349104b859649a0b49aea41af53ff3e P 2772c95187a54a99901b430205a53856 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:17:17.161799 27739 raft_consensus.cc:2454] T a349104b859649a0b49aea41af53ff3e P 90bd992782594257addca235ac0da375 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8085d41a2b5941d0b20614771188c810 in term 1.
I20260620 20:17:17.162285 27806 leader_election.cc:302] T a349104b859649a0b49aea41af53ff3e P 8085d41a2b5941d0b20614771188c810 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8085d41a2b5941d0b20614771188c810, 90bd992782594257addca235ac0da375; no voters:
I20260620 20:17:17.162402 28007 raft_consensus.cc:2454] T a349104b859649a0b49aea41af53ff3e P 2772c95187a54a99901b430205a53856 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8085d41a2b5941d0b20614771188c810 in term 1.
I20260620 20:17:17.162554 28078 raft_consensus.cc:2781] T a349104b859649a0b49aea41af53ff3e P 8085d41a2b5941d0b20614771188c810 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:17:17.163062 28078 raft_consensus.cc:686] T a349104b859649a0b49aea41af53ff3e P 8085d41a2b5941d0b20614771188c810 [term 1 LEADER]: Becoming Leader. State: Replica: 8085d41a2b5941d0b20614771188c810, State: Running, Role: LEADER
I20260620 20:17:17.163609 28078 consensus_queue.cc:237] T a349104b859649a0b49aea41af53ff3e P 8085d41a2b5941d0b20614771188c810 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90bd992782594257addca235ac0da375" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 45147 } } peers { permanent_uuid: "2772c95187a54a99901b430205a53856" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42887 } } peers { permanent_uuid: "8085d41a2b5941d0b20614771188c810" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35297 } }
I20260620 20:17:17.168402 27595 catalog_manager.cc:5495] T a349104b859649a0b49aea41af53ff3e P 8085d41a2b5941d0b20614771188c810 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8085d41a2b5941d0b20614771188c810 (127.16.217.2). New cstate: current_term: 1 leader_uuid: "8085d41a2b5941d0b20614771188c810" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90bd992782594257addca235ac0da375" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 45147 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2772c95187a54a99901b430205a53856" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42887 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8085d41a2b5941d0b20614771188c810" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35297 } health_report { overall_health: HEALTHY } } }
I20260620 20:17:17.186373 28068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:17.180974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40046 (local address 127.16.217.2:35297)
0620 20:17:17.181231 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0620 20:17:17.181238 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:17:17.181285 (+ 47us) server_negotiation.cc:407] Connection header received
0620 20:17:17.181552 (+ 267us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:17:17.181562 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:17:17.181640 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:17:17.181800 (+ 160us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 20:17:17.182196 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:17.182915 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:17.184211 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:17.184610 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:17.184735 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:17:17.185264 (+ 529us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 20:17:17.185757 (+ 493us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 20:17:17.185934 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0620 20:17:17.185994 (+ 60us) server_negotiation.cc:299] Negotiation successful
0620 20:17:17.186066 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
W20260620 20:17:17.190618 27785 tablet.cc:2369] T a349104b859649a0b49aea41af53ff3e P 90bd992782594257addca235ac0da375: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 20:17:17.195676 27739 raft_consensus.cc:1261] T a349104b859649a0b49aea41af53ff3e P 90bd992782594257addca235ac0da375 [term 1 FOLLOWER]: Refusing update from remote peer 8085d41a2b5941d0b20614771188c810: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:17:17.195744 28007 raft_consensus.cc:1261] T a349104b859649a0b49aea41af53ff3e P 2772c95187a54a99901b430205a53856 [term 1 FOLLOWER]: Refusing update from remote peer 8085d41a2b5941d0b20614771188c810: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:17:17.196703 28078 consensus_queue.cc:1035] T a349104b859649a0b49aea41af53ff3e P 8085d41a2b5941d0b20614771188c810 [LEADER]: Connected to new peer: Peer: permanent_uuid: "90bd992782594257addca235ac0da375" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 45147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:17:17.197361 28081 consensus_queue.cc:1035] T a349104b859649a0b49aea41af53ff3e P 8085d41a2b5941d0b20614771188c810 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2772c95187a54a99901b430205a53856" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42887 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:17:17.209247 28084 mvcc.cc:205] Tried to move back new op lower bound from 7299017265941389312 to 7299017265825865728. Current Snapshot: MvccSnapshot[applied={T|T < 7299017265941389312}]
I20260620 20:17:17.214524 28085 mvcc.cc:205] Tried to move back new op lower bound from 7299017265941389312 to 7299017265825865728. Current Snapshot: MvccSnapshot[applied={T|T < 7299017265941389312}]
I20260620 20:17:17.228125 27594 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43820:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 20:17:17.228437 27594 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43820:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 20:17:17.231904 27594 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a5463cecdcd042f18b8c6518b290da62: Sending DeleteTablet for 3 replicas of tablet a349104b859649a0b49aea41af53ff3e
I20260620 20:17:17.233352 27987 tablet_service.cc:1514] Processing DeleteTablet for tablet a349104b859649a0b49aea41af53ff3e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:17:17 UTC) from {username='kudu', principal='kudu/127.16.217.62@KRBTEST.COM'} at 127.0.0.1:38598
I20260620 20:17:17.233515 27853 tablet_service.cc:1514] Processing DeleteTablet for tablet a349104b859649a0b49aea41af53ff3e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:17:17 UTC) from {username='kudu', principal='kudu/127.16.217.62@KRBTEST.COM'} at 127.0.0.1:40040
I20260620 20:17:17.233546 27719 tablet_service.cc:1514] Processing DeleteTablet for tablet a349104b859649a0b49aea41af53ff3e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:17:17 UTC) from {username='kudu', principal='kudu/127.16.217.62@KRBTEST.COM'} at 127.0.0.1:58550
I20260620 20:17:17.234350 17252 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu pid 27656
I20260620 20:17:17.245788 28094 tablet_replica.cc:330] T a349104b859649a0b49aea41af53ff3e P 2772c95187a54a99901b430205a53856: stopping tablet replica
I20260620 20:17:17.246327 28094 raft_consensus.cc:2229] T a349104b859649a0b49aea41af53ff3e P 2772c95187a54a99901b430205a53856 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 20:17:17.246964 28094 raft_consensus.cc:2258] T a349104b859649a0b49aea41af53ff3e P 2772c95187a54a99901b430205a53856 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:17:17.247797 28095 tablet_replica.cc:330] T a349104b859649a0b49aea41af53ff3e P 90bd992782594257addca235ac0da375: stopping tablet replica
I20260620 20:17:17.248270 28095 raft_consensus.cc:2229] T a349104b859649a0b49aea41af53ff3e P 90bd992782594257addca235ac0da375 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 20:17:17.248775 28095 raft_consensus.cc:2258] T a349104b859649a0b49aea41af53ff3e P 90bd992782594257addca235ac0da375 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:17:17.250976 28094 ts_tablet_manager.cc:1905] T a349104b859649a0b49aea41af53ff3e P 2772c95187a54a99901b430205a53856: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:17:17.251356 28095 ts_tablet_manager.cc:1905] T a349104b859649a0b49aea41af53ff3e P 90bd992782594257addca235ac0da375: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:17:17.253460 28096 tablet_replica.cc:330] T a349104b859649a0b49aea41af53ff3e P 8085d41a2b5941d0b20614771188c810: stopping tablet replica
I20260620 20:17:17.253980 28096 raft_consensus.cc:2229] T a349104b859649a0b49aea41af53ff3e P 8085d41a2b5941d0b20614771188c810 [term 1 LEADER]: Raft consensus shutting down.
I20260620 20:17:17.254805 28096 raft_consensus.cc:2258] T a349104b859649a0b49aea41af53ff3e P 8085d41a2b5941d0b20614771188c810 [term 1 FOLLOWER]: Raft consensus is shut down!
Jun 20 20:17:17 dist-test-slave-2hpg krb5kdc[27550](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986634, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.217.1@KRBTEST.COM
I20260620 20:17:17.258481 28095 ts_tablet_manager.cc:1918] T a349104b859649a0b49aea41af53ff3e P 90bd992782594257addca235ac0da375: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 20:17:17.258677 28095 log.cc:1196] T a349104b859649a0b49aea41af53ff3e P 90bd992782594257addca235ac0da375: Deleting WAL directory at /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986536687152-17252-0/minicluster-data/ts-0/wal/wals/a349104b859649a0b49aea41af53ff3e
I20260620 20:17:17.258754 28094 ts_tablet_manager.cc:1918] T a349104b859649a0b49aea41af53ff3e P 2772c95187a54a99901b430205a53856: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 20:17:17.258929 28094 log.cc:1196] T a349104b859649a0b49aea41af53ff3e P 2772c95187a54a99901b430205a53856: Deleting WAL directory at /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986536687152-17252-0/minicluster-data/ts-2/wal/wals/a349104b859649a0b49aea41af53ff3e
I20260620 20:17:17.259268 28095 ts_tablet_manager.cc:1939] T a349104b859649a0b49aea41af53ff3e P 90bd992782594257addca235ac0da375: Deleting consensus metadata
I20260620 20:17:17.259503 28094 ts_tablet_manager.cc:1939] T a349104b859649a0b49aea41af53ff3e P 2772c95187a54a99901b430205a53856: Deleting consensus metadata
I20260620 20:17:17.261272 27579 catalog_manager.cc:4841] TS 90bd992782594257addca235ac0da375 (127.16.217.1:45147): tablet a349104b859649a0b49aea41af53ff3e (table test-table [id=9c1b6098b6b84ad28870fcb04ae3c138]) successfully deleted
I20260620 20:17:17.261991 27580 catalog_manager.cc:4841] TS 2772c95187a54a99901b430205a53856 (127.16.217.3:42887): tablet a349104b859649a0b49aea41af53ff3e (table test-table [id=9c1b6098b6b84ad28870fcb04ae3c138]) successfully deleted
I20260620 20:17:17.262674 28096 ts_tablet_manager.cc:1905] T a349104b859649a0b49aea41af53ff3e P 8085d41a2b5941d0b20614771188c810: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:17:17.265797 28066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:17.235440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58562 (local address 127.16.217.1:45147)
0620 20:17:17.235630 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0620 20:17:17.235672 (+ 42us) server_negotiation.cc:399] Waiting for connection header
0620 20:17:17.246502 (+ 10830us) server_negotiation.cc:407] Connection header received
0620 20:17:17.246875 (+ 373us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:17:17.246883 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:17:17.246984 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:17:17.247074 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:17:17.249053 (+ 1979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:17.250248 (+ 1195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:17.252254 (+ 2006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:17.253101 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:17.253249 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:17:17.258811 (+ 5562us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:17:17.258826 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:17:17.258848 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:17:17.258908 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:17:17.262989 (+ 4081us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:17:17.263778 (+ 789us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:17:17.263785 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:17:17.263788 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:17:17.263914 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:17:17.264265 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:17:17.264271 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:17:17.264274 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:17:17.264602 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:17:17.264893 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0620 20:17:17.265181 (+ 288us) server_negotiation.cc:299] Negotiation successful
0620 20:17:17.265396 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 20:17:17.266448 27779 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:17:17.271093 28096 ts_tablet_manager.cc:1918] T a349104b859649a0b49aea41af53ff3e P 8085d41a2b5941d0b20614771188c810: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 20:17:17.271301 28096 log.cc:1196] T a349104b859649a0b49aea41af53ff3e P 8085d41a2b5941d0b20614771188c810: Deleting WAL directory at /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986536687152-17252-0/minicluster-data/ts-1/wal/wals/a349104b859649a0b49aea41af53ff3e
I20260620 20:17:17.271941 28096 ts_tablet_manager.cc:1939] T a349104b859649a0b49aea41af53ff3e P 8085d41a2b5941d0b20614771188c810: Deleting consensus metadata
I20260620 20:17:17.273999 27579 catalog_manager.cc:4841] TS 8085d41a2b5941d0b20614771188c810 (127.16.217.2:35297): tablet a349104b859649a0b49aea41af53ff3e (table test-table [id=9c1b6098b6b84ad28870fcb04ae3c138]) successfully deleted
I20260620 20:17:17.354667 17252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu with pid 27656
I20260620 20:17:17.373181 17252 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu pid 27790
Jun 20 20:17:17 dist-test-slave-2hpg krb5kdc[27550](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986634, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.217.2@KRBTEST.COM
I20260620 20:17:17.391084 28068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:17.373983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40062 (local address 127.16.217.2:35297)
0620 20:17:17.374178 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0620 20:17:17.374185 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:17:17.374209 (+ 24us) server_negotiation.cc:407] Connection header received
0620 20:17:17.374331 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:17:17.374337 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:17:17.374454 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:17:17.374589 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:17:17.375806 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:17.377046 (+ 1240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:17.378671 (+ 1625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:17.379378 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:17.379521 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:17:17.383713 (+ 4192us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:17:17.383732 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:17:17.383753 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:17:17.383795 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:17:17.387981 (+ 4186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:17:17.388689 (+ 708us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:17:17.388700 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:17:17.388704 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:17:17.388881 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:17:17.389246 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:17:17.389252 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:17:17.389256 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:17:17.389765 (+ 509us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:17:17.390159 (+ 394us) server_negotiation.cc:1035] Waiting for connection context
0620 20:17:17.390367 (+ 208us) server_negotiation.cc:299] Negotiation successful
0620 20:17:17.390618 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260620 20:17:17.391989 27913 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:17:17.501641 17252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu with pid 27790
I20260620 20:17:17.519778 17252 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu pid 27924
Jun 20 20:17:17 dist-test-slave-2hpg krb5kdc[27550](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986634, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.217.3@KRBTEST.COM
I20260620 20:17:17.536048 28067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:17.520539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38612 (local address 127.16.217.3:42887)
0620 20:17:17.520764 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0620 20:17:17.520770 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:17:17.520858 (+ 88us) server_negotiation.cc:407] Connection header received
0620 20:17:17.521108 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:17:17.521135 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:17:17.521213 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:17:17.521334 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:17:17.522845 (+ 1511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:17.523664 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:17.525140 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:17.525561 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:17.525710 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:17:17.530279 (+ 4569us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:17:17.530295 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:17:17.530312 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:17:17.530356 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:17:17.533104 (+ 2748us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:17:17.533847 (+ 743us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:17:17.533855 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:17:17.533857 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:17:17.534052 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:17:17.534434 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:17:17.534440 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:17:17.534444 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:17:17.534858 (+ 414us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:17:17.535159 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0620 20:17:17.535503 (+ 344us) server_negotiation.cc:299] Negotiation successful
0620 20:17:17.535727 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260620 20:17:17.536711 28047 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:17:17.624621 17252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu with pid 27924
I20260620 20:17:17.641196 17252 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu pid 27563
I20260620 20:17:17.642575 27625 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:17:17.748951 17252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu with pid 27563
2026-06-20T20:17:17Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (4192 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986536687152-17252-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 20:17:17 dist-test-slave-2hpg krb5kdc[28104](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 20:17:17 dist-test-slave-2hpg krb5kdc[28104](info): set up 2 sockets
Jun 20 20:17:17 dist-test-slave-2hpg krb5kdc[28104](info): commencing operation
krb5kdc: starting...
W20260620 20:17:18.851281 17252 mini_kdc.cc:121] Time spent starting KDC: real 1.060s user 0.000s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986536687152-17252-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986536687152-17252-0/krb5kdc/test-user.keytab.
Jun 20 20:17:18 dist-test-slave-2hpg krb5kdc[28104](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986638, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T20:17:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T20:17:19Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.16.217.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.16.217.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.16.217.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986536687152-17252-0/krb5kdc/rangeradmin_127.16.217.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.16.217.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986536687152-17252-0/krb5kdc/rangeradmin_127.16.217.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.16.217.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.16.217.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.16.217.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986536687152-17252-0/krb5kdc/rangerlookup_127.16.217.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.16.217.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986536687152-17252-0/krb5kdc/rangerlookup_127.16.217.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.16.217.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.16.217.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.16.217.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986536687152-17252-0/krb5kdc/HTTP_127.16.217.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.16.217.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986536687152-17252-0/krb5kdc/HTTP_127.16.217.20@KRBTEST.COM.keytab.
I20260620 20:17:19.172492 17252 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986536687152-17252-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskQiX1sS/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986536687152-17252-0/minicluster-data/postgres -l logfile start
2026-06-20 20:17:21.675 UTC [28134] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-20 20:17:21.675 UTC [28134] LOG: listening on IPv4 address "127.16.217.20", port 54641
2026-06-20 20:17:21.678 UTC [28134] LOG: listening on Unix socket "/tmp/.s.PGSQL.54641"
2026-06-20 20:17:21.699 UTC [28137] LOG: database system was shut down at 2026-06-20 20:17:20 UTC
2026-06-20 20:17:21.703 UTC [28134] LOG: database system is ready to accept connections
I20260620 20:17:22.678783 17252 mini_postgres.cc:96] Postgres bound to 54641
2026-06-20 20:17:22.696 UTC [28145] FATAL: database "slave" does not exist
127.16.217.20:54641 - accepting connections
I20260620 20:17:22.697134 17252 mini_ranger.cc:161] Starting Ranger...
I20260620 20:17:22.722069 17252 mini_ranger.cc:84] Created miniranger Postgres user
I20260620 20:17:23.200160 17252 mini_ranger.cc:87] Created ranger Postgres database
I20260620 20:17:23.200269 17252 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986536687152-17252-0/minicluster-data/ranger-admin
2026-06-20 20:17:23,495 [I] DB FLAVOR :POSTGRES
2026-06-20 20:17:23,495 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 20:17:23,496 [I] Checking connection..
2026-06-20 20:17:23,496 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQiX1sS/build/asan/bin/postgresql.jar:/tmp/dist-test-taskQiX1sS/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.217.20:54641/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-20 20:17:23,760 [I] Checking connection passed.
2026-06-20 20:17:23,760 [I] --------- Verifying version history table ---------
2026-06-20 20:17:23,760 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQiX1sS/build/asan/bin/postgresql.jar:/tmp/dist-test-taskQiX1sS/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.217.20:54641/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 20:17:24,018 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 20:17:24,019 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQiX1sS/build/asan/bin/postgresql.jar:/tmp/dist-test-taskQiX1sS/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.217.20:54641/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 20:17:24,286 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 20:17:24,286 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-20 20:17:24,286 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQiX1sS/build/asan/bin/postgresql.jar:/tmp/dist-test-taskQiX1sS/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.217.20:54641/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskQiX1sS/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-20 20:17:24.557 UTC [28209] WARNING: there is no transaction in progress
2026-06-20 20:17:24,564 [I] create_dbversion_catalog.sql file imported successfully
2026-06-20 20:17:24,564 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQiX1sS/build/asan/bin/postgresql.jar:/tmp/dist-test-taskQiX1sS/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.217.20:54641/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 20:17:24,807 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-20 20:17:24,808 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 20:17:24,808 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQiX1sS/build/asan/bin/postgresql.jar:/tmp/dist-test-taskQiX1sS/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.217.20:54641/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-20 20:17:25,077 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQiX1sS/build/asan/bin/postgresql.jar:/tmp/dist-test-taskQiX1sS/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.217.20:54641/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-20 20:17:25,331 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQiX1sS/build/asan/bin/postgresql.jar:/tmp/dist-test-taskQiX1sS/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.217.20:54641/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-2hpg.c.gcp-upstream.internal','N') ;"
2026-06-20 20:17:25,571 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQiX1sS/build/asan/bin/postgresql.jar:/tmp/dist-test-taskQiX1sS/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.217.20:54641/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 20:17:25,831 [I] Table x_portal_user does not exist in database ranger
2026-06-20 20:17:25,831 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQiX1sS/build/asan/bin/postgresql.jar:/tmp/dist-test-taskQiX1sS/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.217.20:54641/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 20:17:26,091 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-20 20:17:26,092 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQiX1sS/build/asan/bin/postgresql.jar:/tmp/dist-test-taskQiX1sS/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.217.20:54641/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 20:17:26,350 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQiX1sS/build/asan/bin/postgresql.jar:/tmp/dist-test-taskQiX1sS/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.217.20:54641/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 20:17:26,608 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-20 20:17:26,609 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQiX1sS/build/asan/bin/postgresql.jar:/tmp/dist-test-taskQiX1sS/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.217.20:54641/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskQiX1sS/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-20 20:17:27.577 UTC [28317] WARNING: there is no transaction in progress
2026-06-20 20:17:27.596 UTC [28317] WARNING: there is no transaction in progress
2026-06-20 20:17:27.612 UTC [28317] WARNING: there is no transaction in progress
2026-06-20 20:17:27.627 UTC [28317] WARNING: there is no transaction in progress
2026-06-20 20:17:27.643 UTC [28317] WARNING: there is no transaction in progress
2026-06-20 20:17:27.659 UTC [28317] WARNING: there is no transaction in progress
2026-06-20 20:17:27.715 UTC [28317] WARNING: there is no transaction in progress
2026-06-20 20:17:27.721 UTC [28317] WARNING: there is no transaction in progress
2026-06-20 20:17:27.735 UTC [28317] WARNING: there is no transaction in progress
2026-06-20 20:17:27.745 UTC [28317] WARNING: there is no transaction in progress
2026-06-20 20:17:27.755 UTC [28317] WARNING: there is no transaction in progress
2026-06-20 20:17:27.768 UTC [28317] WARNING: there is no transaction in progress
2026-06-20 20:17:27.777 UTC [28317] WARNING: there is no transaction in progress
2026-06-20 20:17:27.783 UTC [28317] WARNING: there is no transaction in progress
2026-06-20 20:17:27.790 UTC [28317] WARNING: there is no transaction in progress
2026-06-20 20:17:27.796 UTC [28317] WARNING: there is no transaction in progress
2026-06-20 20:17:28,416 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-20 20:17:28,416 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQiX1sS/build/asan/bin/postgresql.jar:/tmp/dist-test-taskQiX1sS/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.217.20:54641/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-20 20:17:28,670 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-20 20:17:28,670 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQiX1sS/build/asan/bin/postgresql.jar:/tmp/dist-test-taskQiX1sS/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.217.20:54641/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 20:17:28,915 [I] Table x_portal_user already exists in database 'ranger'
2026-06-20 20:17:28,915 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQiX1sS/build/asan/bin/postgresql.jar:/tmp/dist-test-taskQiX1sS/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.217.20:54641/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 20:17:29,149 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-20 20:17:29,149 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQiX1sS/build/asan/bin/postgresql.jar:/tmp/dist-test-taskQiX1sS/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.217.20:54641/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 20:17:29,417 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQiX1sS/build/asan/bin/postgresql.jar:/tmp/dist-test-taskQiX1sS/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.217.20:54641/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 20:17:29,658 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQiX1sS/build/asan/bin/postgresql.jar:/tmp/dist-test-taskQiX1sS/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.217.20:54641/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-2hpg.c.gcp-upstream.internal';"
2026-06-20 20:17:29,910 [I] CORE_DB_SCHEMA import status has been updated
2026-06-20 20:17:29,910 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQiX1sS/build/asan/bin/postgresql.jar:/tmp/dist-test-taskQiX1sS/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.217.20:54641/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-20 20:17:30,140 [I] DB_PATCHES have already been applied
I20260620 20:17:30.146428 17252 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986536687152-17252-0/minicluster-data/ranger-admin:/tmp/dist-test-taskQiX1sS/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskQiX1sS/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskQiX1sS/build/asan/bin/postgresql.jar:/tmp/dist-test-taskQiX1sS/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260620 20:17:30.146519 17252 mini_ranger.cc:193] Using host: 127.16.217.20
I20260620 20:17:30.161682 17252 mini_ranger.cc:227] Ranger admin URL: http://127.16.217.20:42729
Jun 20, 2026 8:17:30 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 8:17:30 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 8:17:30 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986536687152-17252-0/minicluster-data/ranger-admin/ews/webapp
Jun 20, 2026 8:17:30 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986536687152-17252-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 20, 2026 8:17:30 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986536687152-17252-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 20, 2026 8:17:31 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986536687152-17252-0/minicluster-data/ranger-admin/ews/webapp].
Jun 20, 2026 8:17:31 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260620 20:17:31.312328 17252 mini_ranger.cc:160] Time spent starting Ranger: real 8.615s user 0.000s sys 0.039s
Jun 20, 2026 8:17:50 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 20, 2026 8:17:50 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Jun 20, 2026 8:17:50 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 20, 2026 8:17:50 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 20, 2026 8:17:50 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 20, 2026 8:17:50 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 20, 2026 8:17:50 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 20, 2026 8:17:50 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 20, 2026 8:17:50 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 20, 2026 8:17:50 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 20, 2026 8:17:50 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 20, 2026 8:17:50 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 20, 2026 8:17:50 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 20, 2026 8:17:50 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 20, 2026 8:17:50 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 20, 2026 8:17:50 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 20, 2026 8:17:50 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 20, 2026 8:17:51 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260620 20:17:52.196477 17252 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.16.217.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.16.217.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.16.217.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986536687152-17252-0/krb5kdc/rangerkms_127.16.217.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.16.217.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986536687152-17252-0/krb5kdc/rangerkms_127.16.217.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.16.217.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.16.217.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.16.217.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986536687152-17252-0/krb5kdc/HTTP_127.16.217.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.16.217.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986536687152-17252-0/krb5kdc/HTTP_127.16.217.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 20 20:17:52 dist-test-slave-2hpg krb5kdc[28104](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986672, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260620 20:17:52.366065 17252 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260620 20:17:52.388168 17252 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260620 20:17:53.305045 17252 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260620 20:17:53.305176 17252 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986536687152-17252-0/minicluster-data/ranger-kms
I20260620 20:17:53.305343 17252 mini_ranger_kms.cc:227] Using postgres at 127.16.217.20:54641
2026-06-20 20:17:53,720 [I] DB FLAVOR :POSTGRES
2026-06-20 20:17:53,720 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 20:17:53,720 [I] Checking connection
2026-06-20 20:17:53,720 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQiX1sS/build/asan/bin/postgresql.jar:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986536687152-17252-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.217.20:54641/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-20 20:17:53,998 [I] connection success
2026-06-20 20:17:53,998 [I] --------- Verifying Ranger DB tables ---------
2026-06-20 20:17:53,998 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-20 20:17:53,998 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQiX1sS/build/asan/bin/postgresql.jar:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986536687152-17252-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.217.20:54641/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-20 20:17:54,232 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-20 20:17:54,232 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 20:17:54,232 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-20 20:17:54,232 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQiX1sS/build/asan/bin/postgresql.jar:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986536687152-17252-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.217.20:54641/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986536687152-17252-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-20 20:17:54,501 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260620 20:17:54.964049 17252 mini_ranger_kms.cc:310] Created kms service
I20260620 20:17:55.233577 17252 mini_ranger_kms.cc:326] Created kudu user
I20260620 20:17:55.281831 17252 mini_ranger_kms.cc:343] Created rangerkms user
Jun 20 20:17:55 dist-test-slave-2hpg krb5kdc[28104](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986672, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.16.217.20@KRBTEST.COM
I20260620 20:17:55.697224 17252 mini_ranger_kms.cc:384] Added ranger policy
I20260620 20:17:55.697544 17252 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986536687152-17252-0/minicluster-data/ranger-kms:/tmp/dist-test-taskQiX1sS/build/asan/bin/postgresql.jar:/tmp/dist-test-taskQiX1sS/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskQiX1sS/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260620 20:17:55.697588 17252 mini_ranger_kms.cc:240] Using host: 127.16.217.20
I20260620 20:17:55.722919 17252 mini_ranger_kms.cc:276] Ranger KMS PID: 28497
I20260620 20:17:55.723088 17252 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.16.217.20:44139
Jun 20, 2026 8:17:56 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 8:17:56 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 8:17:56 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986536687152-17252-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 20, 2026 8:17:56 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986536687152-17252-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 20, 2026 8:17:57 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986536687152-17252-0/minicluster-data/ranger-kms/ews/webapp].
Jun 20, 2026 8:17:57 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260620 20:17:58.014528 17252 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.648s user 0.120s sys 0.317s
I20260620 20:17:58.014802 17252 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260620 20:17:58.015028 17252 mini_ranger_kms.cc:401] 127.16.217.20:44139/kms/v1/keys
Jun 20 20:17:58 dist-test-slave-2hpg krb5kdc[28104](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781986678, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.16.217.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Jun 20 20:17:58 dist-test-slave-2hpg krb5kdc[28104](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986678, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.16.217.20@KRBTEST.COM for HTTP/127.16.217.20@KRBTEST.COM
Jun 20 20:17:58 dist-test-slave-2hpg krb5kdc[28104](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986678, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.16.217.20@KRBTEST.COM for HTTP/127.16.217.20@KRBTEST.COM
Jun 20 20:18:01 dist-test-slave-2hpg krb5kdc[28104](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986681, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.16.217.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.217.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.217.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.62.keytab.
Entry for principal kudu/127.16.217.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.62.keytab.
I20260620 20:18:01.129520 17252 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
/tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986536687152-17252-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986536687152-17252-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986536687152-17252-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986536687152-17252-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.217.62:33813
--webserver_interface=127.16.217.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.62.keytab
--principal=kudu/127.16.217.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.217.20:35257
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.217.62:33813
--ranger_config_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986536687152-17252-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.16.217.20:44139/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986536687152-17252-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986536687152-17252-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986536687152-17252-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:18:01.503573 28548 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:18:01.503901 28548 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:18:01.503989 28548 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:18:01.511544 28548 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:18:01.511669 28548 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:18:01.511716 28548 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:18:01.511765 28548 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:18:01.511812 28548 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:18:01.522348 28548 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.217.20:35257
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986536687152-17252-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986536687152-17252-0/minicluster-data/master-0/wal
--ranger_kms_url=127.16.217.20:44139/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.16.217.62:33813
--ranger_config_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986536687152-17252-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.62.keytab
--principal=kudu/127.16.217.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.217.62:33813
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986536687152-17252-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.217.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986536687152-17252-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:18:01.523627 28548 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:18:01.524884 28548 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:18:01.537216 28554 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:18:01.537170 28556 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:18:01.537187 28553 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:18:01.538472 28548 server_base.cc:1034] running on GCE node
I20260620 20:18:01.539314 28548 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:18:01.540787 28548 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:18:01.541976 28548 hybrid_clock.cc:648] HybridClock initialized: now 1781986681541943 us; error 53 us; skew 500 ppm
I20260620 20:18:01.542377 28548 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:18:01 dist-test-slave-2hpg krb5kdc[28104](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986681, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.217.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:18:01.549537 28548 init.cc:377] Logged in from keytab as kudu/127.16.217.62@KRBTEST.COM (short username kudu)
I20260620 20:18:01.551147 28548 webserver.cc:460] Webserver started at http://127.16.217.62:42051/ using document root <none> and password file <none>
I20260620 20:18:01.551808 28548 fs_manager.cc:359] Metadata directory not provided
I20260620 20:18:01.551946 28548 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:18:01.552250 28548 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 20:18:01 dist-test-slave-2hpg krb5kdc[28104](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986681, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.217.62@KRBTEST.COM for HTTP/127.16.217.20@KRBTEST.COM
I20260620 20:18:01.608917 28548 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986536687152-17252-0/minicluster-data/master-0/data/instance:
uuid: "bbdaabca9b5e49b8a93e4c124c22a476"
format_stamp: "Formatted at 2026-06-20 20:18:01 on dist-test-slave-2hpg"
server_key: "9ccb8e0c70e92c6b44bfb8ce2be20a49"
server_key_iv: "322a972da0dcf0a84aa2544402639ac1"
server_key_version: "kuduclusterkey@0"
I20260620 20:18:01.610028 28548 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986536687152-17252-0/minicluster-data/master-0/wal/instance:
uuid: "bbdaabca9b5e49b8a93e4c124c22a476"
format_stamp: "Formatted at 2026-06-20 20:18:01 on dist-test-slave-2hpg"
server_key: "9ccb8e0c70e92c6b44bfb8ce2be20a49"
server_key_iv: "322a972da0dcf0a84aa2544402639ac1"
server_key_version: "kuduclusterkey@0"
I20260620 20:18:01.617063 28548 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.001s
I20260620 20:18:01.637318 28563 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:18:01.639310 28548 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260620 20:18:01.639536 28548 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986536687152-17252-0/minicluster-data/master-0/data,/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986536687152-17252-0/minicluster-data/master-0/wal
uuid: "bbdaabca9b5e49b8a93e4c124c22a476"
format_stamp: "Formatted at 2026-06-20 20:18:01 on dist-test-slave-2hpg"
server_key: "9ccb8e0c70e92c6b44bfb8ce2be20a49"
server_key_iv: "322a972da0dcf0a84aa2544402639ac1"
server_key_version: "kuduclusterkey@0"
I20260620 20:18:01.639807 28548 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986536687152-17252-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986536687152-17252-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986536687152-17252-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:18:01.661335 28548 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:18:01.665287 28548 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:18:01.665596 28548 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:18:01.685079 28548 rpc_server.cc:307] RPC server started. Bound to: 127.16.217.62:33813
I20260620 20:18:01.685109 28614 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.217.62:33813 every 8 connection(s)
I20260620 20:18:01.686435 28548 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986536687152-17252-0/minicluster-data/master-0/data/info.pb
I20260620 20:18:01.690727 28615 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:18:01.691653 17252 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu as pid 28548
I20260620 20:18:01.691867 17252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986536687152-17252-0/minicluster-data/master-0/wal/instance
I20260620 20:18:01.692678 17252 external_mini_cluster.cc:1428] Setting key b6e1a4265ac306416e9592e401c82063
I20260620 20:18:01.705251 28615 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bbdaabca9b5e49b8a93e4c124c22a476: Bootstrap starting.
Jun 20 20:18:01 dist-test-slave-2hpg krb5kdc[28104](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986681, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.217.62@KRBTEST.COM
I20260620 20:18:01.708796 28615 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bbdaabca9b5e49b8a93e4c124c22a476: Neither blocks nor log segments found. Creating new log.
I20260620 20:18:01.710505 28615 log.cc:824] T 00000000000000000000000000000000 P bbdaabca9b5e49b8a93e4c124c22a476: Log is configured to *not* fsync() on all Append() calls
I20260620 20:18:01.714110 28615 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bbdaabca9b5e49b8a93e4c124c22a476: No bootstrap required, opened a new log
I20260620 20:18:01.716816 28618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:01.694667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46036 (local address 127.16.217.62:33813)
0620 20:18:01.695431 (+ 764us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:01.695443 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:01.695486 (+ 43us) server_negotiation.cc:407] Connection header received
0620 20:18:01.697048 (+ 1562us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:01.697073 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:01.697460 (+ 387us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:01.697790 (+ 330us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:18:01.699312 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:01.700811 (+ 1499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:01.703100 (+ 2289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:01.703773 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:01.703908 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:01.708218 (+ 4310us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:18:01.708226 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:18:01.708241 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:18:01.708276 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:18:01.712111 (+ 3835us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:01.712828 (+ 717us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:01.712838 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:01.712847 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:01.713059 (+ 212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:01.713490 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:01.713495 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:01.713499 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:01.714208 (+ 709us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:18:01.714494 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0620 20:18:01.714964 (+ 470us) server_negotiation.cc:299] Negotiation successful
0620 20:18:01.715374 (+ 410us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":548,"thread_start_us":383,"threads_started":1}
I20260620 20:18:01.724094 28615 raft_consensus.cc:348] T 00000000000000000000000000000000 P bbdaabca9b5e49b8a93e4c124c22a476 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbdaabca9b5e49b8a93e4c124c22a476" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 33813 } }
I20260620 20:18:01.724630 28615 raft_consensus.cc:374] T 00000000000000000000000000000000 P bbdaabca9b5e49b8a93e4c124c22a476 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:18:01.724720 28615 raft_consensus.cc:729] T 00000000000000000000000000000000 P bbdaabca9b5e49b8a93e4c124c22a476 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbdaabca9b5e49b8a93e4c124c22a476, State: Initialized, Role: FOLLOWER
I20260620 20:18:01.725204 28615 consensus_queue.cc:260] T 00000000000000000000000000000000 P bbdaabca9b5e49b8a93e4c124c22a476 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbdaabca9b5e49b8a93e4c124c22a476" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 33813 } }
I20260620 20:18:01.725410 28615 raft_consensus.cc:388] T 00000000000000000000000000000000 P bbdaabca9b5e49b8a93e4c124c22a476 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 20:18:01.725502 28615 raft_consensus.cc:482] T 00000000000000000000000000000000 P bbdaabca9b5e49b8a93e4c124c22a476 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 20:18:01.725625 28615 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bbdaabca9b5e49b8a93e4c124c22a476 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:18:01.727700 28615 raft_consensus.cc:504] T 00000000000000000000000000000000 P bbdaabca9b5e49b8a93e4c124c22a476 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbdaabca9b5e49b8a93e4c124c22a476" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 33813 } }
I20260620 20:18:01.728127 28615 leader_election.cc:302] T 00000000000000000000000000000000 P bbdaabca9b5e49b8a93e4c124c22a476 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bbdaabca9b5e49b8a93e4c124c22a476; no voters:
I20260620 20:18:01.728677 28615 leader_election.cc:288] T 00000000000000000000000000000000 P bbdaabca9b5e49b8a93e4c124c22a476 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 20:18:01.728839 28620 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bbdaabca9b5e49b8a93e4c124c22a476 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:18:01.729537 28620 raft_consensus.cc:686] T 00000000000000000000000000000000 P bbdaabca9b5e49b8a93e4c124c22a476 [term 1 LEADER]: Becoming Leader. State: Replica: bbdaabca9b5e49b8a93e4c124c22a476, State: Running, Role: LEADER
I20260620 20:18:01.730576 28620 consensus_queue.cc:237] T 00000000000000000000000000000000 P bbdaabca9b5e49b8a93e4c124c22a476 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbdaabca9b5e49b8a93e4c124c22a476" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 33813 } }
I20260620 20:18:01.730888 28615 sys_catalog.cc:564] T 00000000000000000000000000000000 P bbdaabca9b5e49b8a93e4c124c22a476 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:18:01.733707 28621 sys_catalog.cc:455] T 00000000000000000000000000000000 P bbdaabca9b5e49b8a93e4c124c22a476 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bbdaabca9b5e49b8a93e4c124c22a476" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbdaabca9b5e49b8a93e4c124c22a476" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 33813 } } }
I20260620 20:18:01.734092 28621 sys_catalog.cc:458] T 00000000000000000000000000000000 P bbdaabca9b5e49b8a93e4c124c22a476 [sys.catalog]: This master's current role is: LEADER
I20260620 20:18:01.734066 28622 sys_catalog.cc:455] T 00000000000000000000000000000000 P bbdaabca9b5e49b8a93e4c124c22a476 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bbdaabca9b5e49b8a93e4c124c22a476. Latest consensus state: current_term: 1 leader_uuid: "bbdaabca9b5e49b8a93e4c124c22a476" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbdaabca9b5e49b8a93e4c124c22a476" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 33813 } } }
I20260620 20:18:01.734333 28622 sys_catalog.cc:458] T 00000000000000000000000000000000 P bbdaabca9b5e49b8a93e4c124c22a476 [sys.catalog]: This master's current role is: LEADER
I20260620 20:18:01.737466 28628 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 20:18:01.742969 28628 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 20:18:01.743647 28615 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986536687152-17252-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260620 20:18:01.760545 28628 catalog_manager.cc:1347] Generated new cluster ID: 8dbb4999a37e4eb8a13a11ab100bee3c
I20260620 20:18:01.760689 28628 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 20:18:01.770927 28628 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 20:18:01.772859 28628 catalog_manager.cc:1504] Loading token signing keys...
I20260620 20:18:01.794085 28628 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bbdaabca9b5e49b8a93e4c124c22a476: Generated new TSK 0
I20260620 20:18:01.795039 28628 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.217.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.217.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.217.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.1.keytab.
Entry for principal kudu/127.16.217.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.1.keytab.
I20260620 20:18:02.564275 17252 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
/tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986536687152-17252-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986536687152-17252-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986536687152-17252-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986536687152-17252-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.16.217.20:44139/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.217.1:0
--local_ip_for_outbound_sockets=127.16.217.1
--webserver_interface=127.16.217.1
--webserver_port=0
--tserver_master_addrs=127.16.217.62:33813
--keytab_file=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.1.keytab
--principal=kudu/127.16.217.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.217.20:35257
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986536687152-17252-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986536687152-17252-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986536687152-17252-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 20:18:02 dist-test-slave-2hpg krb5kdc[28104](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781986682, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.217.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260620 20:18:03.011615 28660 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:18:03.012068 28660 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:18:03.012302 28660 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:18:03.025457 28660 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:18:03.025660 28660 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:18:03.025974 28660 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.217.1
I20260620 20:18:03.042603 28660 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.217.20:35257
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986536687152-17252-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986536687152-17252-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.16.217.20:44139/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.1.keytab
--principal=kudu/127.16.217.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.217.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986536687152-17252-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.217.1
--webserver_port=0
--tserver_master_addrs=127.16.217.62:33813
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.217.1
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986536687152-17252-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:18:03.044597 28660 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:18:03.046481 28660 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:18:03.074213 28671 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:18:03.074188 28670 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:18:03.075572 28673 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:18:03.076151 28660 server_base.cc:1034] running on GCE node
I20260620 20:18:03.076939 28660 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:18:03.078058 28660 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:18:03.079298 28660 hybrid_clock.cc:648] HybridClock initialized: now 1781986683079239 us; error 51 us; skew 500 ppm
I20260620 20:18:03.079667 28660 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:18:03 dist-test-slave-2hpg krb5kdc[28104](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986683, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.217.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:18:03.083736 28660 init.cc:377] Logged in from keytab as kudu/127.16.217.1@KRBTEST.COM (short username kudu)
I20260620 20:18:03.085551 28660 webserver.cc:460] Webserver started at http://127.16.217.1:34071/ using document root <none> and password file <none>
I20260620 20:18:03.086401 28660 fs_manager.cc:359] Metadata directory not provided
I20260620 20:18:03.086560 28660 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:18:03.086964 28660 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 20:18:03 dist-test-slave-2hpg krb5kdc[28104](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986683, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.217.1@KRBTEST.COM for HTTP/127.16.217.20@KRBTEST.COM
I20260620 20:18:03.108402 28660 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986536687152-17252-0/minicluster-data/ts-0/data/instance:
uuid: "8c5924e5eee946c294523626c097e99a"
format_stamp: "Formatted at 2026-06-20 20:18:03 on dist-test-slave-2hpg"
server_key: "e6803e069fcaf17c8327fabb922de60f"
server_key_iv: "3ca459405ad2d23175702cff0191ed97"
server_key_version: "kuduclusterkey@0"
I20260620 20:18:03.109612 28660 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986536687152-17252-0/minicluster-data/ts-0/wal/instance:
uuid: "8c5924e5eee946c294523626c097e99a"
format_stamp: "Formatted at 2026-06-20 20:18:03 on dist-test-slave-2hpg"
server_key: "e6803e069fcaf17c8327fabb922de60f"
server_key_iv: "3ca459405ad2d23175702cff0191ed97"
server_key_version: "kuduclusterkey@0"
I20260620 20:18:03.123047 28660 fs_manager.cc:691] Time spent creating directory manager: real 0.013s user 0.006s sys 0.003s
I20260620 20:18:03.152316 28680 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:18:03.154474 28660 fs_manager.cc:725] Time spent opening block manager: real 0.011s user 0.008s sys 0.000s
I20260620 20:18:03.154763 28660 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986536687152-17252-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986536687152-17252-0/minicluster-data/ts-0/wal
uuid: "8c5924e5eee946c294523626c097e99a"
format_stamp: "Formatted at 2026-06-20 20:18:03 on dist-test-slave-2hpg"
server_key: "e6803e069fcaf17c8327fabb922de60f"
server_key_iv: "3ca459405ad2d23175702cff0191ed97"
server_key_version: "kuduclusterkey@0"
I20260620 20:18:03.155130 28660 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986536687152-17252-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986536687152-17252-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986536687152-17252-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:18:03.204489 28660 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:18:03.210163 28660 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:18:03.210590 28660 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:18:03.218039 28660 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:18:03.225258 28660 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:18:03.225481 28660 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260620 20:18:03.225703 28660 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:18:03.225837 28660 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 20:18:03 dist-test-slave-2hpg krb5kdc[28104](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986682, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.217.62@KRBTEST.COM for HTTP/127.16.217.20@KRBTEST.COM
I20260620 20:18:03.343672 28660 rpc_server.cc:307] RPC server started. Bound to: 127.16.217.1:45773
I20260620 20:18:03.343688 28792 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.217.1:45773 every 8 connection(s)
I20260620 20:18:03.345754 28660 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986536687152-17252-0/minicluster-data/ts-0/data/info.pb
I20260620 20:18:03.354740 17252 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu as pid 28660
I20260620 20:18:03.354960 17252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986536687152-17252-0/minicluster-data/ts-0/wal/instance
I20260620 20:18:03.355816 17252 external_mini_cluster.cc:1428] Setting key ccaa142cb5e0db56a90dd091b807cc25
Jun 20 20:18:03 dist-test-slave-2hpg krb5kdc[28104](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986683, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.217.1@KRBTEST.COM for kudu/127.16.217.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.16.217.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.217.2@KRBTEST.COM" created.
I20260620 20:18:03.394670 28796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:03.350343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.217.1:35661 (local address 127.16.217.62:33813)
0620 20:18:03.359476 (+ 9133us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:03.359493 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:03.359513 (+ 20us) server_negotiation.cc:407] Connection header received
0620 20:18:03.359597 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:03.359605 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:03.359763 (+ 158us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:03.359876 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:18:03.373152 (+ 13276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:03.374691 (+ 1539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:03.377534 (+ 2843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:03.380687 (+ 3153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:03.380839 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:03.385761 (+ 4922us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:18:03.385769 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:18:03.385776 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:18:03.385819 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:18:03.389450 (+ 3631us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:03.390626 (+ 1176us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:03.390633 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:03.390637 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:03.390789 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:03.391417 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:03.391422 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:03.391425 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:03.392278 (+ 853us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:18:03.392408 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0620 20:18:03.394046 (+ 1638us) server_negotiation.cc:299] Negotiation successful
0620 20:18:03.394287 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8995,"thread_start_us":12573,"threads_started":1}
I20260620 20:18:03.396639 28795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:03.349801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:33813 (local address 127.16.217.1:35661)
0620 20:18:03.355974 (+ 6173us) negotiation.cc:107] Waiting for socket to connect
0620 20:18:03.356042 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0620 20:18:03.358616 (+ 2574us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:18:03.370543 (+ 11927us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:18:03.370561 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:18:03.371815 (+ 1254us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:18:03.372882 (+ 1067us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:03.372918 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:03.374913 (+ 1995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:03.374921 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:03.377287 (+ 2366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:03.377322 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:03.381211 (+ 3889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:03.381220 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:03.381598 (+ 378us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:03.382603 (+ 1005us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:18:03.382641 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:18:03.385475 (+ 2834us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:18:03.389641 (+ 4166us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:18:03.389650 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:18:03.389670 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:18:03.390462 (+ 792us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:18:03.390931 (+ 469us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:18:03.390936 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:18:03.390940 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:18:03.391277 (+ 337us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:18:03.392491 (+ 1214us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:18:03.392500 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:18:03.392994 (+ 494us) client_negotiation.cc:769] Sending connection context
0620 20:18:03.393322 (+ 328us) client_negotiation.cc:240] Negotiation successful
0620 20:18:03.395039 (+ 1717us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5779,"thread_start_us":11942,"threads_started":1}
I20260620 20:18:03.399116 28793 heartbeater.cc:347] Connected to a master server at 127.16.217.62:33813
I20260620 20:18:03.399447 28793 heartbeater.cc:464] Registering TS with master...
I20260620 20:18:03.400309 28793 heartbeater.cc:511] Master 127.16.217.62:33813 requested a full tablet report, sending...
I20260620 20:18:03.403046 28580 ts_manager.cc:194] Registered new tserver with Master: 8c5924e5eee946c294523626c097e99a (127.16.217.1:45773)
I20260620 20:18:03.405177 28580 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.217.1@KRBTEST.COM'} at 127.16.217.1:35661
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.217.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.2.keytab.
Entry for principal kudu/127.16.217.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.2.keytab.
I20260620 20:18:03.442875 17252 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
/tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986536687152-17252-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986536687152-17252-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986536687152-17252-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986536687152-17252-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.16.217.20:44139/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.217.2:0
--local_ip_for_outbound_sockets=127.16.217.2
--webserver_interface=127.16.217.2
--webserver_port=0
--tserver_master_addrs=127.16.217.62:33813
--keytab_file=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.2.keytab
--principal=kudu/127.16.217.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.217.20:35257
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986536687152-17252-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986536687152-17252-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986536687152-17252-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 20:18:03 dist-test-slave-2hpg krb5kdc[28104](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986682, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.217.62@KRBTEST.COM for HTTP/127.16.217.20@KRBTEST.COM
W20260620 20:18:03.953298 28799 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:18:03.953853 28799 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:18:03.954098 28799 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:18:03.966933 28799 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:18:03.967209 28799 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:18:03.967475 28799 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.217.2
I20260620 20:18:03.984083 28799 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.217.20:35257
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986536687152-17252-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986536687152-17252-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.16.217.20:44139/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.2.keytab
--principal=kudu/127.16.217.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.217.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986536687152-17252-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.217.2
--webserver_port=0
--tserver_master_addrs=127.16.217.62:33813
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.217.2
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986536687152-17252-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:18:03.986135 28799 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:18:03.988008 28799 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:18:04.027760 28806 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:18:04.030326 28807 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:18:04.034592 28810 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:18:04.036728 28799 server_base.cc:1034] running on GCE node
I20260620 20:18:04.038493 28799 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:18:04.053411 28799 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:18:04.054805 28799 hybrid_clock.cc:648] HybridClock initialized: now 1781986684054733 us; error 67 us; skew 500 ppm
I20260620 20:18:04.055251 28799 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:18:04 dist-test-slave-2hpg krb5kdc[28104](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986684, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.217.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:18:04.060292 28799 init.cc:377] Logged in from keytab as kudu/127.16.217.2@KRBTEST.COM (short username kudu)
I20260620 20:18:04.062301 28799 webserver.cc:460] Webserver started at http://127.16.217.2:46183/ using document root <none> and password file <none>
I20260620 20:18:04.063185 28799 fs_manager.cc:359] Metadata directory not provided
I20260620 20:18:04.063319 28799 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:18:04.063637 28799 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 20:18:04 dist-test-slave-2hpg krb5kdc[28104](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986684, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.217.2@KRBTEST.COM for HTTP/127.16.217.20@KRBTEST.COM
I20260620 20:18:04.083743 28799 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986536687152-17252-0/minicluster-data/ts-1/data/instance:
uuid: "07afdc5e6b5b43a78813f883eb0f7a03"
format_stamp: "Formatted at 2026-06-20 20:18:04 on dist-test-slave-2hpg"
server_key: "bd76e68bb394c0bcb395c5ff5f038c93"
server_key_iv: "68c2a7d7c956384da7053a6487b7a7e9"
server_key_version: "kuduclusterkey@0"
I20260620 20:18:04.085057 28799 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986536687152-17252-0/minicluster-data/ts-1/wal/instance:
uuid: "07afdc5e6b5b43a78813f883eb0f7a03"
format_stamp: "Formatted at 2026-06-20 20:18:04 on dist-test-slave-2hpg"
server_key: "bd76e68bb394c0bcb395c5ff5f038c93"
server_key_iv: "68c2a7d7c956384da7053a6487b7a7e9"
server_key_version: "kuduclusterkey@0"
I20260620 20:18:04.108364 28799 fs_manager.cc:691] Time spent creating directory manager: real 0.023s user 0.018s sys 0.003s
I20260620 20:18:04.142556 28823 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:18:04.147447 28799 fs_manager.cc:725] Time spent opening block manager: real 0.020s user 0.003s sys 0.013s
I20260620 20:18:04.147744 28799 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986536687152-17252-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986536687152-17252-0/minicluster-data/ts-1/wal
uuid: "07afdc5e6b5b43a78813f883eb0f7a03"
format_stamp: "Formatted at 2026-06-20 20:18:04 on dist-test-slave-2hpg"
server_key: "bd76e68bb394c0bcb395c5ff5f038c93"
server_key_iv: "68c2a7d7c956384da7053a6487b7a7e9"
server_key_version: "kuduclusterkey@0"
I20260620 20:18:04.148109 28799 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986536687152-17252-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986536687152-17252-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986536687152-17252-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:18:04.202037 28799 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:18:04.208318 28799 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:18:04.208689 28799 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:18:04.216002 28799 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:18:04.224187 28799 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:18:04.224381 28799 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260620 20:18:04.224553 28799 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:18:04.224645 28799 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:18:04.397817 28799 rpc_server.cc:307] RPC server started. Bound to: 127.16.217.2:40379
I20260620 20:18:04.397848 28935 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.217.2:40379 every 8 connection(s)
I20260620 20:18:04.399176 28799 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986536687152-17252-0/minicluster-data/ts-1/data/info.pb
I20260620 20:18:04.400823 17252 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu as pid 28799
I20260620 20:18:04.400990 17252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986536687152-17252-0/minicluster-data/ts-1/wal/instance
I20260620 20:18:04.401485 17252 external_mini_cluster.cc:1428] Setting key 975ccca199beea9699bfefd57529a6b9
I20260620 20:18:04.409644 28793 heartbeater.cc:503] Master 127.16.217.62:33813 was elected leader, sending a full tablet report...
Jun 20 20:18:04 dist-test-slave-2hpg krb5kdc[28104](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986684, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.217.2@KRBTEST.COM for kudu/127.16.217.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.16.217.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.217.3@KRBTEST.COM" created.
I20260620 20:18:04.435340 28939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:04.405343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:33813 (local address 127.16.217.2:42133)
0620 20:18:04.406373 (+ 1030us) negotiation.cc:107] Waiting for socket to connect
0620 20:18:04.406432 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0620 20:18:04.409817 (+ 3385us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:18:04.415040 (+ 5223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:18:04.415056 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:18:04.415761 (+ 705us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:18:04.416763 (+ 1002us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:04.416794 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:04.418197 (+ 1403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:04.418206 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:04.420540 (+ 2334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:04.420558 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:04.421622 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:04.421631 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:04.421985 (+ 354us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:04.422867 (+ 882us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:18:04.422898 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:18:04.426272 (+ 3374us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:18:04.430699 (+ 4427us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:18:04.430710 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:18:04.430729 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:18:04.431336 (+ 607us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:18:04.431714 (+ 378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:18:04.431718 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:18:04.431722 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:18:04.431966 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:18:04.432639 (+ 673us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:18:04.432647 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:18:04.433092 (+ 445us) client_negotiation.cc:769] Sending connection context
0620 20:18:04.433408 (+ 316us) client_negotiation.cc:240] Negotiation successful
0620 20:18:04.434004 (+ 596us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":640,"thread_start_us":8844,"threads_started":1}
I20260620 20:18:04.435950 28938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:04.403538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.217.2:42133 (local address 127.16.217.62:33813)
0620 20:18:04.414368 (+ 10830us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:04.414376 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:04.414400 (+ 24us) server_negotiation.cc:407] Connection header received
0620 20:18:04.414464 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:04.414471 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:04.414555 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:04.416651 (+ 2096us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:18:04.416959 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:04.418015 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:04.420727 (+ 2712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:04.421366 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:04.421504 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:04.426573 (+ 5069us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:18:04.426581 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:18:04.426588 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:18:04.426635 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:18:04.430503 (+ 3868us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:04.431469 (+ 966us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:04.431474 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:04.431477 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:04.431588 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:04.432142 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:04.432147 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:04.432150 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:04.432515 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:18:04.432623 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0620 20:18:04.433525 (+ 902us) server_negotiation.cc:299] Negotiation successful
0620 20:18:04.433783 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10653,"thread_start_us":10524,"threads_started":1}
I20260620 20:18:04.437582 28936 heartbeater.cc:347] Connected to a master server at 127.16.217.62:33813
I20260620 20:18:04.437966 28936 heartbeater.cc:464] Registering TS with master...
I20260620 20:18:04.438794 28936 heartbeater.cc:511] Master 127.16.217.62:33813 requested a full tablet report, sending...
I20260620 20:18:04.440477 28580 ts_manager.cc:194] Registered new tserver with Master: 07afdc5e6b5b43a78813f883eb0f7a03 (127.16.217.2:40379)
I20260620 20:18:04.441469 28580 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.217.2@KRBTEST.COM'} at 127.16.217.2:42133
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.217.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.3.keytab.
Entry for principal kudu/127.16.217.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.3.keytab.
I20260620 20:18:04.463364 17252 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
/tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986536687152-17252-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986536687152-17252-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986536687152-17252-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986536687152-17252-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.16.217.20:44139/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.217.3:0
--local_ip_for_outbound_sockets=127.16.217.3
--webserver_interface=127.16.217.3
--webserver_port=0
--tserver_master_addrs=127.16.217.62:33813
--keytab_file=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.3.keytab
--principal=kudu/127.16.217.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.217.20:35257
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986536687152-17252-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986536687152-17252-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986536687152-17252-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:18:04.806586 28942 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:18:04.806869 28942 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:18:04.807021 28942 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:18:04.814375 28942 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:18:04.814535 28942 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:18:04.814774 28942 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.217.3
I20260620 20:18:04.825563 28942 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.217.20:35257
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986536687152-17252-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986536687152-17252-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.16.217.20:44139/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.3.keytab
--principal=kudu/127.16.217.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.217.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986536687152-17252-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.217.3
--webserver_port=0
--tserver_master_addrs=127.16.217.62:33813
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.217.3
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986536687152-17252-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:18:04.826920 28942 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:18:04.828259 28942 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:18:04.842087 28947 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:18:04.842123 28950 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:18:04.842051 28948 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:18:04.843302 28942 server_base.cc:1034] running on GCE node
I20260620 20:18:04.843988 28942 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:18:04.844959 28942 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:18:04.846203 28942 hybrid_clock.cc:648] HybridClock initialized: now 1781986684846146 us; error 48 us; skew 500 ppm
I20260620 20:18:04.846537 28942 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:18:04 dist-test-slave-2hpg krb5kdc[28104](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986684, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.217.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:18:04.849570 28942 init.cc:377] Logged in from keytab as kudu/127.16.217.3@KRBTEST.COM (short username kudu)
I20260620 20:18:04.851104 28942 webserver.cc:460] Webserver started at http://127.16.217.3:44075/ using document root <none> and password file <none>
I20260620 20:18:04.851836 28942 fs_manager.cc:359] Metadata directory not provided
I20260620 20:18:04.851984 28942 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:18:04.852306 28942 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 20:18:04 dist-test-slave-2hpg krb5kdc[28104](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986684, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.217.3@KRBTEST.COM for HTTP/127.16.217.20@KRBTEST.COM
I20260620 20:18:04.869419 28942 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986536687152-17252-0/minicluster-data/ts-2/data/instance:
uuid: "3bd2fa1535ac45b2aaaf3575e3ba47c6"
format_stamp: "Formatted at 2026-06-20 20:18:04 on dist-test-slave-2hpg"
server_key: "eb805c88f12cb6833022fbe30f7ab067"
server_key_iv: "e607b106718013e47778ad47f600c3b1"
server_key_version: "kuduclusterkey@0"
I20260620 20:18:04.870558 28942 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986536687152-17252-0/minicluster-data/ts-2/wal/instance:
uuid: "3bd2fa1535ac45b2aaaf3575e3ba47c6"
format_stamp: "Formatted at 2026-06-20 20:18:04 on dist-test-slave-2hpg"
server_key: "eb805c88f12cb6833022fbe30f7ab067"
server_key_iv: "e607b106718013e47778ad47f600c3b1"
server_key_version: "kuduclusterkey@0"
I20260620 20:18:04.878015 28942 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.001s
I20260620 20:18:04.893663 28957 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:18:04.895318 28942 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 20:18:04.895500 28942 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986536687152-17252-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986536687152-17252-0/minicluster-data/ts-2/wal
uuid: "3bd2fa1535ac45b2aaaf3575e3ba47c6"
format_stamp: "Formatted at 2026-06-20 20:18:04 on dist-test-slave-2hpg"
server_key: "eb805c88f12cb6833022fbe30f7ab067"
server_key_iv: "e607b106718013e47778ad47f600c3b1"
server_key_version: "kuduclusterkey@0"
I20260620 20:18:04.895744 28942 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986536687152-17252-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986536687152-17252-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986536687152-17252-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:18:04.915499 28942 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:18:04.920034 28942 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:18:04.920346 28942 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:18:04.921362 28942 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:18:04.922911 28942 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:18:04.922983 28942 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:18:04.923117 28942 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:18:04.923166 28942 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:18:04.959748 28942 rpc_server.cc:307] RPC server started. Bound to: 127.16.217.3:34101
I20260620 20:18:04.959801 29069 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.217.3:34101 every 8 connection(s)
I20260620 20:18:04.961186 28942 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986536687152-17252-0/minicluster-data/ts-2/data/info.pb
I20260620 20:18:04.967058 17252 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu as pid 28942
I20260620 20:18:04.967226 17252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986536687152-17252-0/minicluster-data/ts-2/wal/instance
I20260620 20:18:04.967612 17252 external_mini_cluster.cc:1428] Setting key c1aa76a2db069ca91a08d1c925509a4d
Jun 20 20:18:04 dist-test-slave-2hpg krb5kdc[28104](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986684, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.217.3@KRBTEST.COM for kudu/127.16.217.62@KRBTEST.COM
I20260620 20:18:04.981088 29072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:04.963907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.217.3:46265 (local address 127.16.217.62:33813)
0620 20:18:04.964392 (+ 485us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:04.964396 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:04.965140 (+ 744us) server_negotiation.cc:407] Connection header received
0620 20:18:04.967023 (+ 1883us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:04.967029 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:04.967164 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:04.967294 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:18:04.969335 (+ 2041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:04.970141 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:04.971656 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:04.972080 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:04.972217 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:04.975481 (+ 3264us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:18:04.975486 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:18:04.975489 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:18:04.975518 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:18:04.978076 (+ 2558us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:04.978777 (+ 701us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:04.978784 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:04.978792 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:04.978909 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:04.979353 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:04.979359 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:04.979363 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:04.979628 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:18:04.979827 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0620 20:18:04.980483 (+ 656us) server_negotiation.cc:299] Negotiation successful
0620 20:18:04.980706 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":389,"thread_start_us":263,"threads_started":1}
I20260620 20:18:04.982411 29073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:04.964306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:33813 (local address 127.16.217.3:46265)
0620 20:18:04.964982 (+ 676us) negotiation.cc:107] Waiting for socket to connect
0620 20:18:04.965018 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 20:18:04.966758 (+ 1740us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:18:04.967529 (+ 771us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:18:04.967537 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:18:04.968456 (+ 919us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:18:04.969176 (+ 720us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:04.969198 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:04.970295 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:04.970300 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:04.971482 (+ 1182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:04.971499 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:04.972298 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:04.972307 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:04.972554 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:04.973178 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:18:04.973199 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:18:04.975260 (+ 2061us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:18:04.978227 (+ 2967us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:18:04.978236 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:18:04.978253 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:18:04.978653 (+ 400us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:18:04.979015 (+ 362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:18:04.979021 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:18:04.979025 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:18:04.979240 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:18:04.979797 (+ 557us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:18:04.979805 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:18:04.980150 (+ 345us) client_negotiation.cc:769] Sending connection context
0620 20:18:04.980413 (+ 263us) client_negotiation.cc:240] Negotiation successful
0620 20:18:04.980918 (+ 505us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":459,"thread_start_us":264,"threads_started":1}
I20260620 20:18:04.983919 29070 heartbeater.cc:347] Connected to a master server at 127.16.217.62:33813
I20260620 20:18:04.984220 29070 heartbeater.cc:464] Registering TS with master...
I20260620 20:18:04.984884 29070 heartbeater.cc:511] Master 127.16.217.62:33813 requested a full tablet report, sending...
I20260620 20:18:04.986268 28580 ts_manager.cc:194] Registered new tserver with Master: 3bd2fa1535ac45b2aaaf3575e3ba47c6 (127.16.217.3:34101)
I20260620 20:18:04.987048 28580 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.217.3@KRBTEST.COM'} at 127.16.217.3:46265
I20260620 20:18:04.995486 17252 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 20:18:05.013020 29072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:05.000339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46050 (local address 127.16.217.62:33813)
0620 20:18:05.000565 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:05.000575 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:05.000998 (+ 423us) server_negotiation.cc:407] Connection header received
0620 20:18:05.001175 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:05.001180 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:05.001265 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:05.001362 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:18:05.002919 (+ 1557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:05.003657 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:05.005008 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:05.005458 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:05.005562 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:05.008108 (+ 2546us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:18:05.008113 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:18:05.008117 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:18:05.008154 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:18:05.010769 (+ 2615us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:05.011407 (+ 638us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:05.011411 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:05.011413 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:05.011526 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:05.011874 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:05.011877 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:05.011879 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:05.012136 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:18:05.012312 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0620 20:18:05.012593 (+ 281us) server_negotiation.cc:299] Negotiation successful
0620 20:18:05.012753 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260620 20:18:05.017340 28580 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46050:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 20:18:05.024715 28580 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 20:18:05.059954 29088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:05.048816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36936 (local address 127.16.217.1:45773)
0620 20:18:05.052347 (+ 3531us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:05.052352 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:05.052373 (+ 21us) server_negotiation.cc:407] Connection header received
0620 20:18:05.052479 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:05.052488 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:05.052724 (+ 236us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:05.052977 (+ 253us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:18:05.053745 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:05.055250 (+ 1505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:05.057124 (+ 1874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:05.058609 (+ 1485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:05.058756 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:05.058990 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0620 20:18:05.059466 (+ 476us) server_negotiation.cc:299] Negotiation successful
0620 20:18:05.059626 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3394,"thread_start_us":3219,"threads_started":1}
I20260620 20:18:05.059988 29083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:05.037008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.3:34101 (local address 127.0.0.1:42218)
0620 20:18:05.049859 (+ 12851us) negotiation.cc:107] Waiting for socket to connect
0620 20:18:05.049901 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 20:18:05.050047 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:18:05.052530 (+ 2483us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:18:05.052535 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:18:05.052562 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:18:05.052949 (+ 387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:05.052961 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:05.054958 (+ 1997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:05.054963 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:05.057203 (+ 2240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:05.057216 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:05.058990 (+ 1774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:05.058996 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:05.059166 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:05.059190 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 20:18:05.059271 (+ 81us) client_negotiation.cc:240] Negotiation successful
0620 20:18:05.059510 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12681,"mutex_wait_us":51,"thread_start_us":14298,"threads_started":1}
I20260620 20:18:05.060113 29084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:05.036969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.2:40379 (local address 127.0.0.1:55870)
0620 20:18:05.046842 (+ 9873us) negotiation.cc:107] Waiting for socket to connect
0620 20:18:05.046919 (+ 77us) client_negotiation.cc:174] Beginning negotiation
0620 20:18:05.047156 (+ 237us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:18:05.052047 (+ 4891us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:18:05.052054 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:18:05.052083 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:18:05.052500 (+ 417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:05.052513 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:05.054482 (+ 1969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:05.054490 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:05.057154 (+ 2664us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:05.057169 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:05.059181 (+ 2012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:05.059187 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:05.059357 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:05.059396 (+ 39us) client_negotiation.cc:769] Sending connection context
0620 20:18:05.059487 (+ 91us) client_negotiation.cc:240] Negotiation successful
0620 20:18:05.059697 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9488,"mutex_wait_us":31,"thread_start_us":10447,"threads_started":1}
I20260620 20:18:05.060724 29087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:05.048490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.1:45773 (local address 127.0.0.1:36936)
0620 20:18:05.051085 (+ 2595us) negotiation.cc:107] Waiting for socket to connect
0620 20:18:05.051100 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 20:18:05.051198 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:18:05.053015 (+ 1817us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:18:05.053019 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:18:05.053042 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:18:05.053378 (+ 336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:05.053389 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:05.055384 (+ 1995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:05.055395 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:05.056954 (+ 1559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:05.056968 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:05.058827 (+ 1859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:05.058836 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:05.059085 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:05.059146 (+ 61us) client_negotiation.cc:769] Sending connection context
0620 20:18:05.059279 (+ 133us) client_negotiation.cc:240] Negotiation successful
0620 20:18:05.060506 (+ 1227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2494,"mutex_wait_us":1075,"thread_start_us":2734,"threads_started":1}
I20260620 20:18:05.061376 29085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:05.037113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55870 (local address 127.16.217.2:40379)
0620 20:18:05.050811 (+ 13698us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:05.050824 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:05.050859 (+ 35us) server_negotiation.cc:407] Connection header received
0620 20:18:05.050965 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:05.050974 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:05.051552 (+ 578us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:05.051762 (+ 210us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:18:05.052713 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:05.054368 (+ 1655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:05.057314 (+ 2946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:05.058652 (+ 1338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:05.058757 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:05.058957 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0620 20:18:05.060560 (+ 1603us) server_negotiation.cc:299] Negotiation successful
0620 20:18:05.060803 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13547,"thread_start_us":13437,"threads_started":1}
I20260620 20:18:05.061408 29086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:05.037521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42218 (local address 127.16.217.3:34101)
0620 20:18:05.051943 (+ 14422us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:05.051952 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:05.051984 (+ 32us) server_negotiation.cc:407] Connection header received
0620 20:18:05.052065 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:05.052073 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:05.052322 (+ 249us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:05.052591 (+ 269us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:18:05.053160 (+ 569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:05.054826 (+ 1666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:05.057367 (+ 2541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:05.058780 (+ 1413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:05.060586 (+ 1806us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:05.060809 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0620 20:18:05.060881 (+ 72us) server_negotiation.cc:299] Negotiation successful
0620 20:18:05.061047 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14248,"thread_start_us":14095,"threads_started":1}
I20260620 20:18:05.064694 29005 tablet_service.cc:1467] Processing CreateTablet for tablet be00d10d27e44611a8c0c2f068a56268 (DEFAULT_TABLE table=test-table [id=50ff6592da5d45a2b5bac77ce5aa92d2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:18:05.064718 28728 tablet_service.cc:1467] Processing CreateTablet for tablet be00d10d27e44611a8c0c2f068a56268 (DEFAULT_TABLE table=test-table [id=50ff6592da5d45a2b5bac77ce5aa92d2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:18:05.064718 28871 tablet_service.cc:1467] Processing CreateTablet for tablet be00d10d27e44611a8c0c2f068a56268 (DEFAULT_TABLE table=test-table [id=50ff6592da5d45a2b5bac77ce5aa92d2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:18:05.066030 29005 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be00d10d27e44611a8c0c2f068a56268. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:18:05.066063 28728 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be00d10d27e44611a8c0c2f068a56268. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:18:05.066061 28871 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be00d10d27e44611a8c0c2f068a56268. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:18:05.077255 29089 tablet_bootstrap.cc:492] T be00d10d27e44611a8c0c2f068a56268 P 3bd2fa1535ac45b2aaaf3575e3ba47c6: Bootstrap starting.
I20260620 20:18:05.080531 29089 tablet_bootstrap.cc:654] T be00d10d27e44611a8c0c2f068a56268 P 3bd2fa1535ac45b2aaaf3575e3ba47c6: Neither blocks nor log segments found. Creating new log.
I20260620 20:18:05.081187 29090 tablet_bootstrap.cc:492] T be00d10d27e44611a8c0c2f068a56268 P 07afdc5e6b5b43a78813f883eb0f7a03: Bootstrap starting.
I20260620 20:18:05.082051 29089 log.cc:824] T be00d10d27e44611a8c0c2f068a56268 P 3bd2fa1535ac45b2aaaf3575e3ba47c6: Log is configured to *not* fsync() on all Append() calls
I20260620 20:18:05.082235 29091 tablet_bootstrap.cc:492] T be00d10d27e44611a8c0c2f068a56268 P 8c5924e5eee946c294523626c097e99a: Bootstrap starting.
I20260620 20:18:05.084816 29089 tablet_bootstrap.cc:492] T be00d10d27e44611a8c0c2f068a56268 P 3bd2fa1535ac45b2aaaf3575e3ba47c6: No bootstrap required, opened a new log
I20260620 20:18:05.085080 29089 ts_tablet_manager.cc:1397] T be00d10d27e44611a8c0c2f068a56268 P 3bd2fa1535ac45b2aaaf3575e3ba47c6: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260620 20:18:05.086282 29091 tablet_bootstrap.cc:654] T be00d10d27e44611a8c0c2f068a56268 P 8c5924e5eee946c294523626c097e99a: Neither blocks nor log segments found. Creating new log.
I20260620 20:18:05.086916 29090 tablet_bootstrap.cc:654] T be00d10d27e44611a8c0c2f068a56268 P 07afdc5e6b5b43a78813f883eb0f7a03: Neither blocks nor log segments found. Creating new log.
I20260620 20:18:05.087661 29091 log.cc:824] T be00d10d27e44611a8c0c2f068a56268 P 8c5924e5eee946c294523626c097e99a: Log is configured to *not* fsync() on all Append() calls
I20260620 20:18:05.088160 29090 log.cc:824] T be00d10d27e44611a8c0c2f068a56268 P 07afdc5e6b5b43a78813f883eb0f7a03: Log is configured to *not* fsync() on all Append() calls
I20260620 20:18:05.090840 29091 tablet_bootstrap.cc:492] T be00d10d27e44611a8c0c2f068a56268 P 8c5924e5eee946c294523626c097e99a: No bootstrap required, opened a new log
I20260620 20:18:05.091104 29091 ts_tablet_manager.cc:1397] T be00d10d27e44611a8c0c2f068a56268 P 8c5924e5eee946c294523626c097e99a: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260620 20:18:05.091300 29090 tablet_bootstrap.cc:492] T be00d10d27e44611a8c0c2f068a56268 P 07afdc5e6b5b43a78813f883eb0f7a03: No bootstrap required, opened a new log
I20260620 20:18:05.091532 29090 ts_tablet_manager.cc:1397] T be00d10d27e44611a8c0c2f068a56268 P 07afdc5e6b5b43a78813f883eb0f7a03: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.000s
I20260620 20:18:05.094827 29089 raft_consensus.cc:348] T be00d10d27e44611a8c0c2f068a56268 P 3bd2fa1535ac45b2aaaf3575e3ba47c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07afdc5e6b5b43a78813f883eb0f7a03" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 40379 } } peers { permanent_uuid: "3bd2fa1535ac45b2aaaf3575e3ba47c6" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 34101 } } peers { permanent_uuid: "8c5924e5eee946c294523626c097e99a" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 45773 } }
I20260620 20:18:05.095258 29089 raft_consensus.cc:374] T be00d10d27e44611a8c0c2f068a56268 P 3bd2fa1535ac45b2aaaf3575e3ba47c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:18:05.095403 29089 raft_consensus.cc:729] T be00d10d27e44611a8c0c2f068a56268 P 3bd2fa1535ac45b2aaaf3575e3ba47c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3bd2fa1535ac45b2aaaf3575e3ba47c6, State: Initialized, Role: FOLLOWER
I20260620 20:18:05.096046 29089 consensus_queue.cc:260] T be00d10d27e44611a8c0c2f068a56268 P 3bd2fa1535ac45b2aaaf3575e3ba47c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07afdc5e6b5b43a78813f883eb0f7a03" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 40379 } } peers { permanent_uuid: "3bd2fa1535ac45b2aaaf3575e3ba47c6" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 34101 } } peers { permanent_uuid: "8c5924e5eee946c294523626c097e99a" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 45773 } }
I20260620 20:18:05.097548 29070 heartbeater.cc:503] Master 127.16.217.62:33813 was elected leader, sending a full tablet report...
I20260620 20:18:05.097797 29089 ts_tablet_manager.cc:1428] T be00d10d27e44611a8c0c2f068a56268 P 3bd2fa1535ac45b2aaaf3575e3ba47c6: Time spent starting tablet: real 0.013s user 0.013s sys 0.000s
I20260620 20:18:05.101411 29091 raft_consensus.cc:348] T be00d10d27e44611a8c0c2f068a56268 P 8c5924e5eee946c294523626c097e99a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07afdc5e6b5b43a78813f883eb0f7a03" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 40379 } } peers { permanent_uuid: "3bd2fa1535ac45b2aaaf3575e3ba47c6" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 34101 } } peers { permanent_uuid: "8c5924e5eee946c294523626c097e99a" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 45773 } }
I20260620 20:18:05.101876 29091 raft_consensus.cc:374] T be00d10d27e44611a8c0c2f068a56268 P 8c5924e5eee946c294523626c097e99a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:18:05.102013 29091 raft_consensus.cc:729] T be00d10d27e44611a8c0c2f068a56268 P 8c5924e5eee946c294523626c097e99a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c5924e5eee946c294523626c097e99a, State: Initialized, Role: FOLLOWER
I20260620 20:18:05.102085 29090 raft_consensus.cc:348] T be00d10d27e44611a8c0c2f068a56268 P 07afdc5e6b5b43a78813f883eb0f7a03 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07afdc5e6b5b43a78813f883eb0f7a03" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 40379 } } peers { permanent_uuid: "3bd2fa1535ac45b2aaaf3575e3ba47c6" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 34101 } } peers { permanent_uuid: "8c5924e5eee946c294523626c097e99a" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 45773 } }
I20260620 20:18:05.102483 29090 raft_consensus.cc:374] T be00d10d27e44611a8c0c2f068a56268 P 07afdc5e6b5b43a78813f883eb0f7a03 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:18:05.102587 29090 raft_consensus.cc:729] T be00d10d27e44611a8c0c2f068a56268 P 07afdc5e6b5b43a78813f883eb0f7a03 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07afdc5e6b5b43a78813f883eb0f7a03, State: Initialized, Role: FOLLOWER
I20260620 20:18:05.102627 29091 consensus_queue.cc:260] T be00d10d27e44611a8c0c2f068a56268 P 8c5924e5eee946c294523626c097e99a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07afdc5e6b5b43a78813f883eb0f7a03" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 40379 } } peers { permanent_uuid: "3bd2fa1535ac45b2aaaf3575e3ba47c6" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 34101 } } peers { permanent_uuid: "8c5924e5eee946c294523626c097e99a" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 45773 } }
I20260620 20:18:05.103139 29090 consensus_queue.cc:260] T be00d10d27e44611a8c0c2f068a56268 P 07afdc5e6b5b43a78813f883eb0f7a03 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07afdc5e6b5b43a78813f883eb0f7a03" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 40379 } } peers { permanent_uuid: "3bd2fa1535ac45b2aaaf3575e3ba47c6" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 34101 } } peers { permanent_uuid: "8c5924e5eee946c294523626c097e99a" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 45773 } }
I20260620 20:18:05.104276 29091 ts_tablet_manager.cc:1428] T be00d10d27e44611a8c0c2f068a56268 P 8c5924e5eee946c294523626c097e99a: Time spent starting tablet: real 0.013s user 0.008s sys 0.004s
I20260620 20:18:05.110373 28936 heartbeater.cc:503] Master 127.16.217.62:33813 was elected leader, sending a full tablet report...
I20260620 20:18:05.111362 29090 ts_tablet_manager.cc:1428] T be00d10d27e44611a8c0c2f068a56268 P 07afdc5e6b5b43a78813f883eb0f7a03: Time spent starting tablet: real 0.020s user 0.018s sys 0.000s
W20260620 20:18:05.126044 28794 tablet.cc:2369] T be00d10d27e44611a8c0c2f068a56268 P 8c5924e5eee946c294523626c097e99a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 20:18:05.151237 28937 tablet.cc:2369] T be00d10d27e44611a8c0c2f068a56268 P 07afdc5e6b5b43a78813f883eb0f7a03: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 20:18:05.213073 29071 tablet.cc:2369] T be00d10d27e44611a8c0c2f068a56268 P 3bd2fa1535ac45b2aaaf3575e3ba47c6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 20:18:05.255075 29095 raft_consensus.cc:482] T be00d10d27e44611a8c0c2f068a56268 P 3bd2fa1535ac45b2aaaf3575e3ba47c6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 20:18:05.255355 29095 raft_consensus.cc:504] T be00d10d27e44611a8c0c2f068a56268 P 3bd2fa1535ac45b2aaaf3575e3ba47c6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07afdc5e6b5b43a78813f883eb0f7a03" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 40379 } } peers { permanent_uuid: "3bd2fa1535ac45b2aaaf3575e3ba47c6" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 34101 } } peers { permanent_uuid: "8c5924e5eee946c294523626c097e99a" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 45773 } }
I20260620 20:18:05.256918 29095 leader_election.cc:288] T be00d10d27e44611a8c0c2f068a56268 P 3bd2fa1535ac45b2aaaf3575e3ba47c6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 07afdc5e6b5b43a78813f883eb0f7a03 (127.16.217.2:40379), 8c5924e5eee946c294523626c097e99a (127.16.217.1:45773)
I20260620 20:18:05.264163 29085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:05.257381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.217.3:52295 (local address 127.16.217.2:40379)
0620 20:18:05.257605 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:05.257611 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:05.257738 (+ 127us) server_negotiation.cc:407] Connection header received
0620 20:18:05.258025 (+ 287us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:05.258031 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:05.258104 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:05.258191 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:18:05.258890 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:05.259972 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:05.262427 (+ 2455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:05.263295 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:05.263459 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:05.263531 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 20:18:05.263781 (+ 250us) server_negotiation.cc:299] Negotiation successful
0620 20:18:05.263913 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260620 20:18:05.264217 29073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:05.257408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.2:40379 (local address 127.16.217.3:52295)
0620 20:18:05.257644 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0620 20:18:05.257665 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 20:18:05.257823 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:18:05.258395 (+ 572us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:18:05.258401 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:18:05.258434 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:18:05.258645 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:05.258661 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:05.260148 (+ 1487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:05.260153 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:05.262249 (+ 2096us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:05.262263 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:05.263451 (+ 1188us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:05.263456 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:05.263629 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:05.263653 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 20:18:05.263758 (+ 105us) client_negotiation.cc:240] Negotiation successful
0620 20:18:05.263901 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56,"mutex_wait_us":4}
I20260620 20:18:05.264643 29088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:05.258079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.217.3:47463 (local address 127.16.217.1:45773)
0620 20:18:05.258361 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:05.258368 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:05.258780 (+ 412us) server_negotiation.cc:407] Connection header received
0620 20:18:05.258946 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:05.258966 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:05.259069 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:05.259186 (+ 117us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:18:05.259696 (+ 510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:05.260665 (+ 969us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:05.262907 (+ 2242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:05.263742 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:05.263919 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:05.263989 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 20:18:05.264269 (+ 280us) server_negotiation.cc:299] Negotiation successful
0620 20:18:05.264394 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260620 20:18:05.264726 29098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:05.257917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.1:45773 (local address 127.16.217.3:47463)
0620 20:18:05.258665 (+ 748us) negotiation.cc:107] Waiting for socket to connect
0620 20:18:05.258681 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 20:18:05.258805 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:18:05.259218 (+ 413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:18:05.259221 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:18:05.259243 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:18:05.259520 (+ 277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:05.259527 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:05.260858 (+ 1331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:05.260865 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:05.262749 (+ 1884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:05.262759 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:05.263948 (+ 1189us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:05.263954 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:05.264106 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:05.264135 (+ 29us) client_negotiation.cc:769] Sending connection context
0620 20:18:05.264244 (+ 109us) client_negotiation.cc:240] Negotiation successful
0620 20:18:05.264363 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":629,"thread_start_us":388,"threads_started":1}
I20260620 20:18:05.265450 28891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be00d10d27e44611a8c0c2f068a56268" candidate_uuid: "3bd2fa1535ac45b2aaaf3575e3ba47c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07afdc5e6b5b43a78813f883eb0f7a03" is_pre_election: true
I20260620 20:18:05.265816 28748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be00d10d27e44611a8c0c2f068a56268" candidate_uuid: "3bd2fa1535ac45b2aaaf3575e3ba47c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c5924e5eee946c294523626c097e99a" is_pre_election: true
I20260620 20:18:05.265990 28891 raft_consensus.cc:2454] T be00d10d27e44611a8c0c2f068a56268 P 07afdc5e6b5b43a78813f883eb0f7a03 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3bd2fa1535ac45b2aaaf3575e3ba47c6 in term 0.
I20260620 20:18:05.266309 28748 raft_consensus.cc:2454] T be00d10d27e44611a8c0c2f068a56268 P 8c5924e5eee946c294523626c097e99a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3bd2fa1535ac45b2aaaf3575e3ba47c6 in term 0.
I20260620 20:18:05.266670 28960 leader_election.cc:302] T be00d10d27e44611a8c0c2f068a56268 P 3bd2fa1535ac45b2aaaf3575e3ba47c6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07afdc5e6b5b43a78813f883eb0f7a03, 3bd2fa1535ac45b2aaaf3575e3ba47c6; no voters:
I20260620 20:18:05.267023 29095 raft_consensus.cc:2781] T be00d10d27e44611a8c0c2f068a56268 P 3bd2fa1535ac45b2aaaf3575e3ba47c6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 20:18:05.267146 29095 raft_consensus.cc:482] T be00d10d27e44611a8c0c2f068a56268 P 3bd2fa1535ac45b2aaaf3575e3ba47c6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 20:18:05.267266 29095 raft_consensus.cc:3037] T be00d10d27e44611a8c0c2f068a56268 P 3bd2fa1535ac45b2aaaf3575e3ba47c6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:18:05.269440 29095 raft_consensus.cc:504] T be00d10d27e44611a8c0c2f068a56268 P 3bd2fa1535ac45b2aaaf3575e3ba47c6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07afdc5e6b5b43a78813f883eb0f7a03" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 40379 } } peers { permanent_uuid: "3bd2fa1535ac45b2aaaf3575e3ba47c6" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 34101 } } peers { permanent_uuid: "8c5924e5eee946c294523626c097e99a" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 45773 } }
I20260620 20:18:05.270104 29095 leader_election.cc:288] T be00d10d27e44611a8c0c2f068a56268 P 3bd2fa1535ac45b2aaaf3575e3ba47c6 [CANDIDATE]: Term 1 election: Requested vote from peers 07afdc5e6b5b43a78813f883eb0f7a03 (127.16.217.2:40379), 8c5924e5eee946c294523626c097e99a (127.16.217.1:45773)
I20260620 20:18:05.270556 28891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be00d10d27e44611a8c0c2f068a56268" candidate_uuid: "3bd2fa1535ac45b2aaaf3575e3ba47c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07afdc5e6b5b43a78813f883eb0f7a03"
I20260620 20:18:05.270665 28748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be00d10d27e44611a8c0c2f068a56268" candidate_uuid: "3bd2fa1535ac45b2aaaf3575e3ba47c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c5924e5eee946c294523626c097e99a"
I20260620 20:18:05.270777 28891 raft_consensus.cc:3037] T be00d10d27e44611a8c0c2f068a56268 P 07afdc5e6b5b43a78813f883eb0f7a03 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:18:05.270893 28748 raft_consensus.cc:3037] T be00d10d27e44611a8c0c2f068a56268 P 8c5924e5eee946c294523626c097e99a [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:18:05.274044 28748 raft_consensus.cc:2454] T be00d10d27e44611a8c0c2f068a56268 P 8c5924e5eee946c294523626c097e99a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3bd2fa1535ac45b2aaaf3575e3ba47c6 in term 1.
I20260620 20:18:05.274044 28891 raft_consensus.cc:2454] T be00d10d27e44611a8c0c2f068a56268 P 07afdc5e6b5b43a78813f883eb0f7a03 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3bd2fa1535ac45b2aaaf3575e3ba47c6 in term 1.
I20260620 20:18:05.274466 28960 leader_election.cc:302] T be00d10d27e44611a8c0c2f068a56268 P 3bd2fa1535ac45b2aaaf3575e3ba47c6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07afdc5e6b5b43a78813f883eb0f7a03, 3bd2fa1535ac45b2aaaf3575e3ba47c6; no voters:
I20260620 20:18:05.274796 29095 raft_consensus.cc:2781] T be00d10d27e44611a8c0c2f068a56268 P 3bd2fa1535ac45b2aaaf3575e3ba47c6 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:18:05.275333 29095 raft_consensus.cc:686] T be00d10d27e44611a8c0c2f068a56268 P 3bd2fa1535ac45b2aaaf3575e3ba47c6 [term 1 LEADER]: Becoming Leader. State: Replica: 3bd2fa1535ac45b2aaaf3575e3ba47c6, State: Running, Role: LEADER
I20260620 20:18:05.275858 29095 consensus_queue.cc:237] T be00d10d27e44611a8c0c2f068a56268 P 3bd2fa1535ac45b2aaaf3575e3ba47c6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07afdc5e6b5b43a78813f883eb0f7a03" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 40379 } } peers { permanent_uuid: "3bd2fa1535ac45b2aaaf3575e3ba47c6" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 34101 } } peers { permanent_uuid: "8c5924e5eee946c294523626c097e99a" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 45773 } }
I20260620 20:18:05.280458 28579 catalog_manager.cc:5495] T be00d10d27e44611a8c0c2f068a56268 P 3bd2fa1535ac45b2aaaf3575e3ba47c6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3bd2fa1535ac45b2aaaf3575e3ba47c6 (127.16.217.3). New cstate: current_term: 1 leader_uuid: "3bd2fa1535ac45b2aaaf3575e3ba47c6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07afdc5e6b5b43a78813f883eb0f7a03" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 40379 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3bd2fa1535ac45b2aaaf3575e3ba47c6" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 34101 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8c5924e5eee946c294523626c097e99a" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 45773 } health_report { overall_health: UNKNOWN } } }
I20260620 20:18:05.345184 29086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:05.339007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42224 (local address 127.16.217.3:34101)
0620 20:18:05.339187 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:05.339213 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:05.339254 (+ 41us) server_negotiation.cc:407] Connection header received
0620 20:18:05.339368 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:05.339376 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:05.339482 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:05.339630 (+ 148us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 20:18:05.340008 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:05.340767 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:05.342671 (+ 1904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:05.343268 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:05.343392 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:05.343907 (+ 515us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 20:18:05.344601 (+ 694us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 20:18:05.344748 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0620 20:18:05.344850 (+ 102us) server_negotiation.cc:299] Negotiation successful
0620 20:18:05.344931 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 20:18:05.355017 28748 raft_consensus.cc:1261] T be00d10d27e44611a8c0c2f068a56268 P 8c5924e5eee946c294523626c097e99a [term 1 FOLLOWER]: Refusing update from remote peer 3bd2fa1535ac45b2aaaf3575e3ba47c6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:18:05.355437 28891 raft_consensus.cc:1261] T be00d10d27e44611a8c0c2f068a56268 P 07afdc5e6b5b43a78813f883eb0f7a03 [term 1 FOLLOWER]: Refusing update from remote peer 3bd2fa1535ac45b2aaaf3575e3ba47c6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:18:05.356141 29095 consensus_queue.cc:1035] T be00d10d27e44611a8c0c2f068a56268 P 3bd2fa1535ac45b2aaaf3575e3ba47c6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c5924e5eee946c294523626c097e99a" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 45773 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:18:05.356648 29099 consensus_queue.cc:1035] T be00d10d27e44611a8c0c2f068a56268 P 3bd2fa1535ac45b2aaaf3575e3ba47c6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "07afdc5e6b5b43a78813f883eb0f7a03" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 40379 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:18:05.381983 28580 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46050:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 20:18:05.382278 28580 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46050:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 20:18:05.385762 28580 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bbdaabca9b5e49b8a93e4c124c22a476: Sending DeleteTablet for 3 replicas of tablet be00d10d27e44611a8c0c2f068a56268
I20260620 20:18:05.386916 28871 tablet_service.cc:1514] Processing DeleteTablet for tablet be00d10d27e44611a8c0c2f068a56268 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:18:05 UTC) from {username='kudu', principal='kudu/127.16.217.62@KRBTEST.COM'} at 127.0.0.1:55870
I20260620 20:18:05.386956 28728 tablet_service.cc:1514] Processing DeleteTablet for tablet be00d10d27e44611a8c0c2f068a56268 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:18:05 UTC) from {username='kudu', principal='kudu/127.16.217.62@KRBTEST.COM'} at 127.0.0.1:36936
I20260620 20:18:05.386952 29005 tablet_service.cc:1514] Processing DeleteTablet for tablet be00d10d27e44611a8c0c2f068a56268 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:18:05 UTC) from {username='kudu', principal='kudu/127.16.217.62@KRBTEST.COM'} at 127.0.0.1:42218
I20260620 20:18:05.390165 17252 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu pid 28660
I20260620 20:18:05.415545 29113 tablet_replica.cc:330] T be00d10d27e44611a8c0c2f068a56268 P 8c5924e5eee946c294523626c097e99a: stopping tablet replica
I20260620 20:18:05.416047 29113 raft_consensus.cc:2229] T be00d10d27e44611a8c0c2f068a56268 P 8c5924e5eee946c294523626c097e99a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 20:18:05.416602 29113 raft_consensus.cc:2258] T be00d10d27e44611a8c0c2f068a56268 P 8c5924e5eee946c294523626c097e99a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:18:05.417330 29112 tablet_replica.cc:330] T be00d10d27e44611a8c0c2f068a56268 P 07afdc5e6b5b43a78813f883eb0f7a03: stopping tablet replica
I20260620 20:18:05.417903 29112 raft_consensus.cc:2229] T be00d10d27e44611a8c0c2f068a56268 P 07afdc5e6b5b43a78813f883eb0f7a03 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 20:18:05.418447 29112 raft_consensus.cc:2258] T be00d10d27e44611a8c0c2f068a56268 P 07afdc5e6b5b43a78813f883eb0f7a03 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:18:05.421118 29112 ts_tablet_manager.cc:1905] T be00d10d27e44611a8c0c2f068a56268 P 07afdc5e6b5b43a78813f883eb0f7a03: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:18:05.424105 29114 tablet_replica.cc:330] T be00d10d27e44611a8c0c2f068a56268 P 3bd2fa1535ac45b2aaaf3575e3ba47c6: stopping tablet replica
I20260620 20:18:05.424925 29114 raft_consensus.cc:2229] T be00d10d27e44611a8c0c2f068a56268 P 3bd2fa1535ac45b2aaaf3575e3ba47c6 [term 1 LEADER]: Raft consensus shutting down.
I20260620 20:18:05.425256 29113 ts_tablet_manager.cc:1905] T be00d10d27e44611a8c0c2f068a56268 P 8c5924e5eee946c294523626c097e99a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:18:05.425702 29114 raft_consensus.cc:2258] T be00d10d27e44611a8c0c2f068a56268 P 3bd2fa1535ac45b2aaaf3575e3ba47c6 [term 1 FOLLOWER]: Raft consensus is shut down!
Jun 20 20:18:05 dist-test-slave-2hpg krb5kdc[28104](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986681, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.217.1@KRBTEST.COM
I20260620 20:18:05.427567 29114 ts_tablet_manager.cc:1905] T be00d10d27e44611a8c0c2f068a56268 P 3bd2fa1535ac45b2aaaf3575e3ba47c6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:18:05.432636 29113 ts_tablet_manager.cc:1918] T be00d10d27e44611a8c0c2f068a56268 P 8c5924e5eee946c294523626c097e99a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 20:18:05.432891 29113 log.cc:1196] T be00d10d27e44611a8c0c2f068a56268 P 8c5924e5eee946c294523626c097e99a: Deleting WAL directory at /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986536687152-17252-0/minicluster-data/ts-0/wal/wals/be00d10d27e44611a8c0c2f068a56268
I20260620 20:18:05.433501 29113 ts_tablet_manager.cc:1939] T be00d10d27e44611a8c0c2f068a56268 P 8c5924e5eee946c294523626c097e99a: Deleting consensus metadata
I20260620 20:18:05.435341 29114 ts_tablet_manager.cc:1918] T be00d10d27e44611a8c0c2f068a56268 P 3bd2fa1535ac45b2aaaf3575e3ba47c6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 20:18:05.435602 29114 log.cc:1196] T be00d10d27e44611a8c0c2f068a56268 P 3bd2fa1535ac45b2aaaf3575e3ba47c6: Deleting WAL directory at /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986536687152-17252-0/minicluster-data/ts-2/wal/wals/be00d10d27e44611a8c0c2f068a56268
I20260620 20:18:05.436234 29114 ts_tablet_manager.cc:1939] T be00d10d27e44611a8c0c2f068a56268 P 3bd2fa1535ac45b2aaaf3575e3ba47c6: Deleting consensus metadata
I20260620 20:18:05.436262 28566 catalog_manager.cc:4841] TS 8c5924e5eee946c294523626c097e99a (127.16.217.1:45773): tablet be00d10d27e44611a8c0c2f068a56268 (table test-table [id=50ff6592da5d45a2b5bac77ce5aa92d2]) successfully deleted
I20260620 20:18:05.437947 28565 catalog_manager.cc:4841] TS 3bd2fa1535ac45b2aaaf3575e3ba47c6 (127.16.217.3:34101): tablet be00d10d27e44611a8c0c2f068a56268 (table test-table [id=50ff6592da5d45a2b5bac77ce5aa92d2]) successfully deleted
I20260620 20:18:05.440810 29088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:05.392123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36940 (local address 127.16.217.1:45773)
0620 20:18:05.392314 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:05.392320 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:05.415645 (+ 23325us) server_negotiation.cc:407] Connection header received
0620 20:18:05.416384 (+ 739us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:05.416390 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:05.416471 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:05.416545 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:18:05.418851 (+ 2306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:05.420024 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:05.422072 (+ 2048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:05.422690 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:05.422781 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:05.432537 (+ 9756us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:18:05.432552 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:18:05.432570 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:18:05.432617 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:18:05.437266 (+ 4649us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:05.438326 (+ 1060us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:05.438333 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:05.438336 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:05.438462 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:05.438879 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:05.438886 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:05.438889 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:05.439312 (+ 423us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:18:05.439581 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0620 20:18:05.440107 (+ 526us) server_negotiation.cc:299] Negotiation successful
0620 20:18:05.440323 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260620 20:18:05.441764 29112 ts_tablet_manager.cc:1918] T be00d10d27e44611a8c0c2f068a56268 P 07afdc5e6b5b43a78813f883eb0f7a03: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 20:18:05.441952 29112 log.cc:1196] T be00d10d27e44611a8c0c2f068a56268 P 07afdc5e6b5b43a78813f883eb0f7a03: Deleting WAL directory at /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986536687152-17252-0/minicluster-data/ts-1/wal/wals/be00d10d27e44611a8c0c2f068a56268
I20260620 20:18:05.442477 29112 ts_tablet_manager.cc:1939] T be00d10d27e44611a8c0c2f068a56268 P 07afdc5e6b5b43a78813f883eb0f7a03: Deleting consensus metadata
I20260620 20:18:05.443415 28788 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:18:05.444711 28566 catalog_manager.cc:4841] TS 07afdc5e6b5b43a78813f883eb0f7a03 (127.16.217.2:40379): tablet be00d10d27e44611a8c0c2f068a56268 (table test-table [id=50ff6592da5d45a2b5bac77ce5aa92d2]) successfully deleted
I20260620 20:18:05.552807 17252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu with pid 28660
I20260620 20:18:05.572104 17252 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu pid 28799
Jun 20 20:18:05 dist-test-slave-2hpg krb5kdc[28104](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986681, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.217.2@KRBTEST.COM
I20260620 20:18:05.590415 29085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:05.572994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55876 (local address 127.16.217.2:40379)
0620 20:18:05.573211 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:05.573218 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:05.573241 (+ 23us) server_negotiation.cc:407] Connection header received
0620 20:18:05.573310 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:05.573315 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:05.573405 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:05.573515 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:18:05.574799 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:05.575934 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:05.577186 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:05.577828 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:05.578017 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:05.582766 (+ 4749us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:18:05.582788 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:18:05.582814 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:18:05.582863 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:18:05.586966 (+ 4103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:05.588083 (+ 1117us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:05.588093 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:05.588097 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:05.588265 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:05.588669 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:05.588675 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:05.588678 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:05.589162 (+ 484us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:18:05.589460 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0620 20:18:05.589722 (+ 262us) server_negotiation.cc:299] Negotiation successful
0620 20:18:05.590004 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 20:18:05.591230 28931 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:18:05.680909 17252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu with pid 28799
I20260620 20:18:05.703147 17252 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu pid 28942
Jun 20 20:18:05 dist-test-slave-2hpg krb5kdc[28104](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986681, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.217.3@KRBTEST.COM
I20260620 20:18:05.721571 29086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:05.703953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42236 (local address 127.16.217.3:34101)
0620 20:18:05.704154 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:05.704160 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:05.704184 (+ 24us) server_negotiation.cc:407] Connection header received
0620 20:18:05.704316 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:05.704323 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:05.704425 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:05.704548 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:18:05.705999 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:05.707187 (+ 1188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:05.708518 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:05.709226 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:05.709381 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:05.714217 (+ 4836us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:18:05.714237 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:18:05.714265 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:18:05.714317 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:18:05.718405 (+ 4088us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:05.719228 (+ 823us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:05.719240 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:05.719244 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:05.719411 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:05.719873 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:05.719878 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:05.719881 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:05.720367 (+ 486us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:18:05.720693 (+ 326us) server_negotiation.cc:1035] Waiting for connection context
0620 20:18:05.720931 (+ 238us) server_negotiation.cc:299] Negotiation successful
0620 20:18:05.721165 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260620 20:18:05.722447 29065 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:18:05.843565 17252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu with pid 28942
I20260620 20:18:05.862810 17252 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu pid 28548
I20260620 20:18:05.863981 28610 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:18:05.978322 17252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu with pid 28548
I20260620 20:18:06.000962 17252 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260620 20:18:06.336772 17252 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260620 20:18:06.337018 17252 mini_ranger.cc:66] Stopping Ranger...
I20260620 20:18:06.683140 17252 mini_ranger.cc:68] Stopped Ranger
2026-06-20 20:18:06.683 UTC [28134] LOG: received smart shutdown request
2026-06-20 20:18:06.686 UTC [28134] LOG: background worker "logical replication launcher" (PID 28143) exited with exit code 1
2026-06-20 20:18:06.687 UTC [28138] LOG: shutting down
2026-06-20 20:18:06.839 UTC [28134] LOG: database system is shut down
2026-06-20T20:18:06Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (49232 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986536687152-17252-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 20:18:07 dist-test-slave-2hpg krb5kdc[29128](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 20:18:07 dist-test-slave-2hpg krb5kdc[29128](info): set up 2 sockets
Jun 20 20:18:07 dist-test-slave-2hpg krb5kdc[29128](info): commencing operation
krb5kdc: starting...
W20260620 20:18:08.090556 17252 mini_kdc.cc:121] Time spent starting KDC: real 1.066s user 0.000s sys 0.030s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986536687152-17252-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986536687152-17252-0/krb5kdc/test-user.keytab.
Jun 20 20:18:08 dist-test-slave-2hpg krb5kdc[29128](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986688, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T20:18:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T20:18:08Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.16.217.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.16.217.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.16.217.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986536687152-17252-0/krb5kdc/rangeradmin_127.16.217.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.16.217.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986536687152-17252-0/krb5kdc/rangeradmin_127.16.217.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.16.217.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.16.217.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.16.217.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986536687152-17252-0/krb5kdc/rangerlookup_127.16.217.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.16.217.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986536687152-17252-0/krb5kdc/rangerlookup_127.16.217.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.16.217.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.16.217.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.16.217.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986536687152-17252-0/krb5kdc/HTTP_127.16.217.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.16.217.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986536687152-17252-0/krb5kdc/HTTP_127.16.217.20@KRBTEST.COM.keytab.
I20260620 20:18:08.419171 17252 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986536687152-17252-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskQiX1sS/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986536687152-17252-0/minicluster-data/postgres -l logfile start
2026-06-20 20:18:10.905 UTC [29158] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-20 20:18:10.905 UTC [29158] LOG: listening on IPv4 address "127.16.217.20", port 60265
2026-06-20 20:18:10.909 UTC [29158] LOG: listening on Unix socket "/tmp/.s.PGSQL.60265"
2026-06-20 20:18:10.930 UTC [29161] LOG: database system was shut down at 2026-06-20 20:18:09 UTC
2026-06-20 20:18:10.934 UTC [29158] LOG: database system is ready to accept connections
I20260620 20:18:11.902348 17252 mini_postgres.cc:96] Postgres bound to 60265
2026-06-20 20:18:11.919 UTC [29169] FATAL: database "slave" does not exist
127.16.217.20:60265 - accepting connections
I20260620 20:18:11.920485 17252 mini_ranger.cc:161] Starting Ranger...
I20260620 20:18:11.946442 17252 mini_ranger.cc:84] Created miniranger Postgres user
I20260620 20:18:12.432739 17252 mini_ranger.cc:87] Created ranger Postgres database
I20260620 20:18:12.432869 17252 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986536687152-17252-0/minicluster-data/ranger-admin
2026-06-20 20:18:12,727 [I] DB FLAVOR :POSTGRES
2026-06-20 20:18:12,727 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 20:18:12,727 [I] Checking connection..
2026-06-20 20:18:12,727 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQiX1sS/build/asan/bin/postgresql.jar:/tmp/dist-test-taskQiX1sS/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.217.20:60265/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-20 20:18:12,975 [I] Checking connection passed.
2026-06-20 20:18:12,976 [I] --------- Verifying version history table ---------
2026-06-20 20:18:12,976 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQiX1sS/build/asan/bin/postgresql.jar:/tmp/dist-test-taskQiX1sS/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.217.20:60265/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 20:18:13,222 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 20:18:13,223 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQiX1sS/build/asan/bin/postgresql.jar:/tmp/dist-test-taskQiX1sS/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.217.20:60265/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 20:18:13,481 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 20:18:13,481 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-20 20:18:13,481 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQiX1sS/build/asan/bin/postgresql.jar:/tmp/dist-test-taskQiX1sS/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.217.20:60265/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskQiX1sS/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-20 20:18:13.729 UTC [29233] WARNING: there is no transaction in progress
2026-06-20 20:18:13,736 [I] create_dbversion_catalog.sql file imported successfully
2026-06-20 20:18:13,736 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQiX1sS/build/asan/bin/postgresql.jar:/tmp/dist-test-taskQiX1sS/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.217.20:60265/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 20:18:13,979 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-20 20:18:13,979 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 20:18:13,979 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQiX1sS/build/asan/bin/postgresql.jar:/tmp/dist-test-taskQiX1sS/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.217.20:60265/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-20 20:18:14,229 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQiX1sS/build/asan/bin/postgresql.jar:/tmp/dist-test-taskQiX1sS/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.217.20:60265/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-20 20:18:14,485 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQiX1sS/build/asan/bin/postgresql.jar:/tmp/dist-test-taskQiX1sS/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.217.20:60265/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-2hpg.c.gcp-upstream.internal','N') ;"
2026-06-20 20:18:14,746 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQiX1sS/build/asan/bin/postgresql.jar:/tmp/dist-test-taskQiX1sS/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.217.20:60265/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 20:18:14,978 [I] Table x_portal_user does not exist in database ranger
2026-06-20 20:18:14,978 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQiX1sS/build/asan/bin/postgresql.jar:/tmp/dist-test-taskQiX1sS/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.217.20:60265/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 20:18:15,224 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-20 20:18:15,224 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQiX1sS/build/asan/bin/postgresql.jar:/tmp/dist-test-taskQiX1sS/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.217.20:60265/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 20:18:15,498 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQiX1sS/build/asan/bin/postgresql.jar:/tmp/dist-test-taskQiX1sS/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.217.20:60265/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 20:18:15,757 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-20 20:18:15,757 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQiX1sS/build/asan/bin/postgresql.jar:/tmp/dist-test-taskQiX1sS/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.217.20:60265/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskQiX1sS/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-20 20:18:16.747 UTC [29341] WARNING: there is no transaction in progress
2026-06-20 20:18:16.764 UTC [29341] WARNING: there is no transaction in progress
2026-06-20 20:18:16.782 UTC [29341] WARNING: there is no transaction in progress
2026-06-20 20:18:16.798 UTC [29341] WARNING: there is no transaction in progress
2026-06-20 20:18:16.815 UTC [29341] WARNING: there is no transaction in progress
2026-06-20 20:18:16.835 UTC [29341] WARNING: there is no transaction in progress
2026-06-20 20:18:16.893 UTC [29341] WARNING: there is no transaction in progress
2026-06-20 20:18:16.900 UTC [29341] WARNING: there is no transaction in progress
2026-06-20 20:18:16.916 UTC [29341] WARNING: there is no transaction in progress
2026-06-20 20:18:16.926 UTC [29341] WARNING: there is no transaction in progress
2026-06-20 20:18:16.938 UTC [29341] WARNING: there is no transaction in progress
2026-06-20 20:18:16.953 UTC [29341] WARNING: there is no transaction in progress
2026-06-20 20:18:16.963 UTC [29341] WARNING: there is no transaction in progress
2026-06-20 20:18:16.970 UTC [29341] WARNING: there is no transaction in progress
2026-06-20 20:18:16.977 UTC [29341] WARNING: there is no transaction in progress
2026-06-20 20:18:16.983 UTC [29341] WARNING: there is no transaction in progress
2026-06-20 20:18:17,647 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-20 20:18:17,648 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQiX1sS/build/asan/bin/postgresql.jar:/tmp/dist-test-taskQiX1sS/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.217.20:60265/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-20 20:18:17,881 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-20 20:18:17,881 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQiX1sS/build/asan/bin/postgresql.jar:/tmp/dist-test-taskQiX1sS/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.217.20:60265/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 20:18:18,142 [I] Table x_portal_user already exists in database 'ranger'
2026-06-20 20:18:18,142 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQiX1sS/build/asan/bin/postgresql.jar:/tmp/dist-test-taskQiX1sS/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.217.20:60265/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 20:18:18,375 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-20 20:18:18,375 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQiX1sS/build/asan/bin/postgresql.jar:/tmp/dist-test-taskQiX1sS/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.217.20:60265/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 20:18:18,641 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQiX1sS/build/asan/bin/postgresql.jar:/tmp/dist-test-taskQiX1sS/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.217.20:60265/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 20:18:18,905 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQiX1sS/build/asan/bin/postgresql.jar:/tmp/dist-test-taskQiX1sS/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.217.20:60265/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-2hpg.c.gcp-upstream.internal';"
2026-06-20 20:18:19,160 [I] CORE_DB_SCHEMA import status has been updated
2026-06-20 20:18:19,161 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQiX1sS/build/asan/bin/postgresql.jar:/tmp/dist-test-taskQiX1sS/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.217.20:60265/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-20 20:18:19,421 [I] DB_PATCHES have already been applied
I20260620 20:18:19.427471 17252 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986536687152-17252-0/minicluster-data/ranger-admin:/tmp/dist-test-taskQiX1sS/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskQiX1sS/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskQiX1sS/build/asan/bin/postgresql.jar:/tmp/dist-test-taskQiX1sS/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260620 20:18:19.427557 17252 mini_ranger.cc:193] Using host: 127.16.217.20
I20260620 20:18:19.442835 17252 mini_ranger.cc:227] Ranger admin URL: http://127.16.217.20:57411
Jun 20, 2026 8:18:19 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 8:18:19 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 8:18:19 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986536687152-17252-0/minicluster-data/ranger-admin/ews/webapp
Jun 20, 2026 8:18:19 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986536687152-17252-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 20, 2026 8:18:19 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986536687152-17252-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 20, 2026 8:18:20 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986536687152-17252-0/minicluster-data/ranger-admin/ews/webapp].
Jun 20, 2026 8:18:20 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260620 20:18:20.569591 17252 mini_ranger.cc:160] Time spent starting Ranger: real 8.649s user 0.005s sys 0.034s
Jun 20, 2026 8:18:40 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 20, 2026 8:18:40 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Jun 20, 2026 8:18:40 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 20, 2026 8:18:40 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 20, 2026 8:18:40 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 20, 2026 8:18:40 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 20, 2026 8:18:40 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 20, 2026 8:18:40 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 20, 2026 8:18:40 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 20, 2026 8:18:40 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 20, 2026 8:18:40 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 20, 2026 8:18:40 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 20, 2026 8:18:40 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 20, 2026 8:18:40 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 20, 2026 8:18:40 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 20, 2026 8:18:40 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 20, 2026 8:18:40 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 20, 2026 8:18:41 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260620 20:18:42.434814 17252 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.16.217.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.16.217.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.16.217.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986536687152-17252-0/krb5kdc/rangerkms_127.16.217.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.16.217.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986536687152-17252-0/krb5kdc/rangerkms_127.16.217.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.16.217.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.16.217.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.16.217.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986536687152-17252-0/krb5kdc/HTTP_127.16.217.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.16.217.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986536687152-17252-0/krb5kdc/HTTP_127.16.217.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 20 20:18:42 dist-test-slave-2hpg krb5kdc[29128](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986722, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260620 20:18:42.623114 17252 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260620 20:18:42.644150 17252 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260620 20:18:43.469501 17252 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260620 20:18:43.469619 17252 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986536687152-17252-0/minicluster-data/ranger-kms
I20260620 20:18:43.469663 17252 mini_ranger_kms.cc:227] Using postgres at 127.16.217.20:60265
2026-06-20 20:18:43,819 [I] DB FLAVOR :POSTGRES
2026-06-20 20:18:43,819 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 20:18:43,819 [I] Checking connection
2026-06-20 20:18:43,819 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQiX1sS/build/asan/bin/postgresql.jar:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986536687152-17252-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.217.20:60265/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-20 20:18:44,088 [I] connection success
2026-06-20 20:18:44,089 [I] --------- Verifying Ranger DB tables ---------
2026-06-20 20:18:44,089 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-20 20:18:44,089 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQiX1sS/build/asan/bin/postgresql.jar:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986536687152-17252-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.217.20:60265/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-20 20:18:44,322 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-20 20:18:44,322 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 20:18:44,323 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-20 20:18:44,323 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQiX1sS/build/asan/bin/postgresql.jar:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986536687152-17252-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.217.20:60265/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986536687152-17252-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-20 20:18:44,569 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260620 20:18:45.037051 17252 mini_ranger_kms.cc:310] Created kms service
I20260620 20:18:45.253458 17252 mini_ranger_kms.cc:326] Created kudu user
I20260620 20:18:45.303524 17252 mini_ranger_kms.cc:343] Created rangerkms user
Jun 20 20:18:45 dist-test-slave-2hpg krb5kdc[29128](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986722, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.16.217.20@KRBTEST.COM
I20260620 20:18:45.735399 17252 mini_ranger_kms.cc:384] Added ranger policy
I20260620 20:18:45.735714 17252 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986536687152-17252-0/minicluster-data/ranger-kms:/tmp/dist-test-taskQiX1sS/build/asan/bin/postgresql.jar:/tmp/dist-test-taskQiX1sS/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskQiX1sS/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260620 20:18:45.735760 17252 mini_ranger_kms.cc:240] Using host: 127.16.217.20
I20260620 20:18:45.756026 17252 mini_ranger_kms.cc:276] Ranger KMS PID: 29521
I20260620 20:18:45.756165 17252 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.16.217.20:49279
Jun 20, 2026 8:18:46 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 8:18:46 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 8:18:46 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986536687152-17252-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 20, 2026 8:18:46 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986536687152-17252-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 20, 2026 8:18:47 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986536687152-17252-0/minicluster-data/ranger-kms/ews/webapp].
Jun 20, 2026 8:18:47 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260620 20:18:48.061247 17252 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.438s user 0.089s sys 0.281s
I20260620 20:18:48.061502 17252 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260620 20:18:48.061684 17252 mini_ranger_kms.cc:401] 127.16.217.20:49279/kms/v1/keys
Jun 20 20:18:48 dist-test-slave-2hpg krb5kdc[29128](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781986728, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.16.217.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Jun 20 20:18:48 dist-test-slave-2hpg krb5kdc[29128](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986728, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.16.217.20@KRBTEST.COM for HTTP/127.16.217.20@KRBTEST.COM
Jun 20 20:18:49 dist-test-slave-2hpg krb5kdc[29128](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986728, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.16.217.20@KRBTEST.COM for HTTP/127.16.217.20@KRBTEST.COM
Jun 20 20:18:51 dist-test-slave-2hpg krb5kdc[29128](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986731, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.16.217.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.217.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.217.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.62.keytab.
Entry for principal kudu/127.16.217.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.62.keytab.
I20260620 20:18:51.317342 17252 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
/tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986536687152-17252-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986536687152-17252-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986536687152-17252-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986536687152-17252-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.217.62:40353
--webserver_interface=127.16.217.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.62.keytab
--principal=kudu/127.16.217.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.217.20:35163
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.217.62:40353
--ranger_config_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986536687152-17252-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.16.217.20:49279/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986536687152-17252-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986536687152-17252-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986536687152-17252-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:18:51.658468 29572 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:18:51.658811 29572 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:18:51.658991 29572 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:18:51.666712 29572 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:18:51.666828 29572 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:18:51.666905 29572 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:18:51.667018 29572 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:18:51.667093 29572 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:18:51.677850 29572 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.217.20:35163
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986536687152-17252-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986536687152-17252-0/minicluster-data/master-0/wal
--ranger_kms_url=127.16.217.20:49279/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.16.217.62:40353
--ranger_config_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986536687152-17252-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.62.keytab
--principal=kudu/127.16.217.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.217.62:40353
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986536687152-17252-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.217.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986536687152-17252-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:18:51.679381 29572 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:18:51.680893 29572 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:18:51.695048 29577 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:18:51.695093 29578 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:18:51.695012 29580 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:18:51.695753 29572 server_base.cc:1034] running on GCE node
I20260620 20:18:51.696606 29572 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:18:51.698112 29572 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:18:51.699368 29572 hybrid_clock.cc:648] HybridClock initialized: now 1781986731699292 us; error 64 us; skew 500 ppm
I20260620 20:18:51.699774 29572 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:18:51 dist-test-slave-2hpg krb5kdc[29128](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986731, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.217.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:18:51.706887 29572 init.cc:377] Logged in from keytab as kudu/127.16.217.62@KRBTEST.COM (short username kudu)
I20260620 20:18:51.708587 29572 webserver.cc:460] Webserver started at http://127.16.217.62:46209/ using document root <none> and password file <none>
I20260620 20:18:51.709261 29572 fs_manager.cc:359] Metadata directory not provided
I20260620 20:18:51.709477 29572 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:18:51.709800 29572 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 20:18:51 dist-test-slave-2hpg krb5kdc[29128](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986731, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.217.62@KRBTEST.COM for HTTP/127.16.217.20@KRBTEST.COM
I20260620 20:18:51.767859 29572 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986536687152-17252-0/minicluster-data/master-0/data/instance:
uuid: "12eb0dcc6c7f4782be00f8df729c5aba"
format_stamp: "Formatted at 2026-06-20 20:18:51 on dist-test-slave-2hpg"
server_key: "c88d4877ff2d8a3b71b8addddcd83823"
server_key_iv: "ab29dc8ff0c7f6b4840a5b299c75f748"
server_key_version: "kuduclusterkey@0"
I20260620 20:18:51.769045 29572 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986536687152-17252-0/minicluster-data/master-0/wal/instance:
uuid: "12eb0dcc6c7f4782be00f8df729c5aba"
format_stamp: "Formatted at 2026-06-20 20:18:51 on dist-test-slave-2hpg"
server_key: "c88d4877ff2d8a3b71b8addddcd83823"
server_key_iv: "ab29dc8ff0c7f6b4840a5b299c75f748"
server_key_version: "kuduclusterkey@0"
I20260620 20:18:51.776074 29572 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.008s sys 0.000s
I20260620 20:18:51.796180 29587 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:18:51.798135 29572 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260620 20:18:51.798345 29572 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986536687152-17252-0/minicluster-data/master-0/data,/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986536687152-17252-0/minicluster-data/master-0/wal
uuid: "12eb0dcc6c7f4782be00f8df729c5aba"
format_stamp: "Formatted at 2026-06-20 20:18:51 on dist-test-slave-2hpg"
server_key: "c88d4877ff2d8a3b71b8addddcd83823"
server_key_iv: "ab29dc8ff0c7f6b4840a5b299c75f748"
server_key_version: "kuduclusterkey@0"
I20260620 20:18:51.798615 29572 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986536687152-17252-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986536687152-17252-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986536687152-17252-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:18:51.845676 29572 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:18:51.851203 29572 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:18:51.851599 29572 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:18:51.873070 29572 rpc_server.cc:307] RPC server started. Bound to: 127.16.217.62:40353
I20260620 20:18:51.873030 29638 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.217.62:40353 every 8 connection(s)
I20260620 20:18:51.874971 29572 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986536687152-17252-0/minicluster-data/master-0/data/info.pb
I20260620 20:18:51.879140 29639 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:18:51.882822 17252 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu as pid 29572
I20260620 20:18:51.883054 17252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986536687152-17252-0/minicluster-data/master-0/wal/instance
I20260620 20:18:51.883932 17252 external_mini_cluster.cc:1428] Setting key e2a7625dd507a0115b9287f7f6f21209
I20260620 20:18:51.893384 29639 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 12eb0dcc6c7f4782be00f8df729c5aba: Bootstrap starting.
I20260620 20:18:51.898607 29639 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 12eb0dcc6c7f4782be00f8df729c5aba: Neither blocks nor log segments found. Creating new log.
I20260620 20:18:51.900102 29639 log.cc:824] T 00000000000000000000000000000000 P 12eb0dcc6c7f4782be00f8df729c5aba: Log is configured to *not* fsync() on all Append() calls
I20260620 20:18:51.903631 29639 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 12eb0dcc6c7f4782be00f8df729c5aba: No bootstrap required, opened a new log
Jun 20 20:18:51 dist-test-slave-2hpg krb5kdc[29128](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986731, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.217.62@KRBTEST.COM
I20260620 20:18:51.913491 29639 raft_consensus.cc:348] T 00000000000000000000000000000000 P 12eb0dcc6c7f4782be00f8df729c5aba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12eb0dcc6c7f4782be00f8df729c5aba" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 40353 } }
I20260620 20:18:51.914597 29639 raft_consensus.cc:374] T 00000000000000000000000000000000 P 12eb0dcc6c7f4782be00f8df729c5aba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:18:51.914799 29639 raft_consensus.cc:729] T 00000000000000000000000000000000 P 12eb0dcc6c7f4782be00f8df729c5aba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12eb0dcc6c7f4782be00f8df729c5aba, State: Initialized, Role: FOLLOWER
I20260620 20:18:51.915625 29639 consensus_queue.cc:260] T 00000000000000000000000000000000 P 12eb0dcc6c7f4782be00f8df729c5aba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12eb0dcc6c7f4782be00f8df729c5aba" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 40353 } }
I20260620 20:18:51.915975 29639 raft_consensus.cc:388] T 00000000000000000000000000000000 P 12eb0dcc6c7f4782be00f8df729c5aba [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 20:18:51.916216 29639 raft_consensus.cc:482] T 00000000000000000000000000000000 P 12eb0dcc6c7f4782be00f8df729c5aba [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 20:18:51.916415 29639 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 12eb0dcc6c7f4782be00f8df729c5aba [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:18:51.919755 29639 raft_consensus.cc:504] T 00000000000000000000000000000000 P 12eb0dcc6c7f4782be00f8df729c5aba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12eb0dcc6c7f4782be00f8df729c5aba" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 40353 } }
I20260620 20:18:51.920353 29639 leader_election.cc:302] T 00000000000000000000000000000000 P 12eb0dcc6c7f4782be00f8df729c5aba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 12eb0dcc6c7f4782be00f8df729c5aba; no voters:
I20260620 20:18:51.921080 29639 leader_election.cc:288] T 00000000000000000000000000000000 P 12eb0dcc6c7f4782be00f8df729c5aba [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 20:18:51.921233 29644 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 12eb0dcc6c7f4782be00f8df729c5aba [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:18:51.926460 29642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:51.886245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60180 (local address 127.16.217.62:40353)
0620 20:18:51.898490 (+ 12245us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:51.898514 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:51.898934 (+ 420us) server_negotiation.cc:407] Connection header received
0620 20:18:51.901504 (+ 2570us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:51.901527 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:51.902071 (+ 544us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:51.902510 (+ 439us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:18:51.904204 (+ 1694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:51.906431 (+ 2227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:51.908477 (+ 2046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:51.909628 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:51.909822 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:51.915094 (+ 5272us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:18:51.915105 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:18:51.915124 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:18:51.915174 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:18:51.919224 (+ 4050us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:51.920105 (+ 881us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:51.920114 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:51.920122 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:51.920310 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:51.920761 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:51.920769 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:51.920775 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:51.921759 (+ 984us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:18:51.923574 (+ 1815us) server_negotiation.cc:1035] Waiting for connection context
0620 20:18:51.923944 (+ 370us) server_negotiation.cc:299] Negotiation successful
0620 20:18:51.924546 (+ 602us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11960,"thread_start_us":11726,"threads_started":1}
I20260620 20:18:51.928196 29644 raft_consensus.cc:686] T 00000000000000000000000000000000 P 12eb0dcc6c7f4782be00f8df729c5aba [term 1 LEADER]: Becoming Leader. State: Replica: 12eb0dcc6c7f4782be00f8df729c5aba, State: Running, Role: LEADER
I20260620 20:18:51.928788 29639 sys_catalog.cc:564] T 00000000000000000000000000000000 P 12eb0dcc6c7f4782be00f8df729c5aba [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:18:51.928869 29644 consensus_queue.cc:237] T 00000000000000000000000000000000 P 12eb0dcc6c7f4782be00f8df729c5aba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12eb0dcc6c7f4782be00f8df729c5aba" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 40353 } }
I20260620 20:18:51.932502 29645 sys_catalog.cc:455] T 00000000000000000000000000000000 P 12eb0dcc6c7f4782be00f8df729c5aba [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "12eb0dcc6c7f4782be00f8df729c5aba" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12eb0dcc6c7f4782be00f8df729c5aba" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 40353 } } }
I20260620 20:18:51.933162 29645 sys_catalog.cc:458] T 00000000000000000000000000000000 P 12eb0dcc6c7f4782be00f8df729c5aba [sys.catalog]: This master's current role is: LEADER
I20260620 20:18:51.932701 29646 sys_catalog.cc:455] T 00000000000000000000000000000000 P 12eb0dcc6c7f4782be00f8df729c5aba [sys.catalog]: SysCatalogTable state changed. Reason: New leader 12eb0dcc6c7f4782be00f8df729c5aba. Latest consensus state: current_term: 1 leader_uuid: "12eb0dcc6c7f4782be00f8df729c5aba" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12eb0dcc6c7f4782be00f8df729c5aba" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 40353 } } }
I20260620 20:18:51.933652 29646 sys_catalog.cc:458] T 00000000000000000000000000000000 P 12eb0dcc6c7f4782be00f8df729c5aba [sys.catalog]: This master's current role is: LEADER
I20260620 20:18:51.950776 29649 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 20:18:51.956741 29649 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 20:18:51.957521 29639 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986536687152-17252-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260620 20:18:51.981913 29649 catalog_manager.cc:1347] Generated new cluster ID: 41e1c0e688564eed8e9496071fd3729e
I20260620 20:18:51.982275 29649 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 20:18:52.003885 29649 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 20:18:52.006142 29649 catalog_manager.cc:1504] Loading token signing keys...
I20260620 20:18:52.014389 29649 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 12eb0dcc6c7f4782be00f8df729c5aba: Generated new TSK 0
I20260620 20:18:52.015686 29649 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.217.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.217.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.217.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.1.keytab.
Entry for principal kudu/127.16.217.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.1.keytab.
I20260620 20:18:52.873554 17252 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
/tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986536687152-17252-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986536687152-17252-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986536687152-17252-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986536687152-17252-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.16.217.20:49279/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.217.1:0
--local_ip_for_outbound_sockets=127.16.217.1
--webserver_interface=127.16.217.1
--webserver_port=0
--tserver_master_addrs=127.16.217.62:40353
--keytab_file=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.1.keytab
--principal=kudu/127.16.217.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.217.20:35163
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986536687152-17252-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986536687152-17252-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986536687152-17252-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 20:18:53 dist-test-slave-2hpg krb5kdc[29128](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781986733, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.217.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260620 20:18:53.258267 29684 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:18:53.258652 29684 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:18:53.258827 29684 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:18:53.267176 29684 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:18:53.267408 29684 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:18:53.267608 29684 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.217.1
I20260620 20:18:53.281013 29684 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.217.20:35163
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986536687152-17252-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986536687152-17252-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.16.217.20:49279/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.1.keytab
--principal=kudu/127.16.217.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.217.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986536687152-17252-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.217.1
--webserver_port=0
--tserver_master_addrs=127.16.217.62:40353
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.217.1
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986536687152-17252-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:18:53.282495 29684 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:18:53.283896 29684 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:18:53.302101 29694 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:18:53.304702 29695 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:18:53.316047 29697 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:18:53.316545 29684 server_base.cc:1034] running on GCE node
I20260620 20:18:53.317389 29684 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:18:53.326160 29684 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:18:53.329660 29684 hybrid_clock.cc:648] HybridClock initialized: now 1781986733329533 us; error 110 us; skew 500 ppm
I20260620 20:18:53.330197 29684 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:18:53 dist-test-slave-2hpg krb5kdc[29128](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986733, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.217.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:18:53.334612 29684 init.cc:377] Logged in from keytab as kudu/127.16.217.1@KRBTEST.COM (short username kudu)
I20260620 20:18:53.336702 29684 webserver.cc:460] Webserver started at http://127.16.217.1:43495/ using document root <none> and password file <none>
I20260620 20:18:53.337661 29684 fs_manager.cc:359] Metadata directory not provided
I20260620 20:18:53.337800 29684 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:18:53.338223 29684 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 20:18:53 dist-test-slave-2hpg krb5kdc[29128](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986733, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.217.1@KRBTEST.COM for HTTP/127.16.217.20@KRBTEST.COM
I20260620 20:18:53.362507 29684 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986536687152-17252-0/minicluster-data/ts-0/data/instance:
uuid: "890d1d3512c14e0288ac4e0ce29ce66d"
format_stamp: "Formatted at 2026-06-20 20:18:53 on dist-test-slave-2hpg"
server_key: "d0fcf224af91cc0ee838039aa68e21b7"
server_key_iv: "1edf6d5edcf0a07d33b1f794585d6fd5"
server_key_version: "kuduclusterkey@0"
I20260620 20:18:53.363798 29684 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986536687152-17252-0/minicluster-data/ts-0/wal/instance:
uuid: "890d1d3512c14e0288ac4e0ce29ce66d"
format_stamp: "Formatted at 2026-06-20 20:18:53 on dist-test-slave-2hpg"
server_key: "d0fcf224af91cc0ee838039aa68e21b7"
server_key_iv: "1edf6d5edcf0a07d33b1f794585d6fd5"
server_key_version: "kuduclusterkey@0"
I20260620 20:18:53.377452 29684 fs_manager.cc:691] Time spent creating directory manager: real 0.013s user 0.003s sys 0.008s
I20260620 20:18:53.414496 29704 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:18:53.416262 29684 fs_manager.cc:725] Time spent opening block manager: real 0.020s user 0.004s sys 0.012s
I20260620 20:18:53.416473 29684 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986536687152-17252-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986536687152-17252-0/minicluster-data/ts-0/wal
uuid: "890d1d3512c14e0288ac4e0ce29ce66d"
format_stamp: "Formatted at 2026-06-20 20:18:53 on dist-test-slave-2hpg"
server_key: "d0fcf224af91cc0ee838039aa68e21b7"
server_key_iv: "1edf6d5edcf0a07d33b1f794585d6fd5"
server_key_version: "kuduclusterkey@0"
I20260620 20:18:53.416714 29684 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986536687152-17252-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986536687152-17252-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986536687152-17252-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:18:53.446521 29684 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:18:53.452266 29684 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:18:53.452674 29684 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:18:53.454586 29684 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:18:53.456465 29684 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:18:53.456602 29684 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260620 20:18:53.456764 29684 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:18:53.456842 29684 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:18:53.526203 29684 rpc_server.cc:307] RPC server started. Bound to: 127.16.217.1:38219
I20260620 20:18:53.526242 29816 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.217.1:38219 every 8 connection(s)
I20260620 20:18:53.527943 29684 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986536687152-17252-0/minicluster-data/ts-0/data/info.pb
I20260620 20:18:53.535341 17252 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu as pid 29684
I20260620 20:18:53.535602 17252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986536687152-17252-0/minicluster-data/ts-0/wal/instance
I20260620 20:18:53.536571 17252 external_mini_cluster.cc:1428] Setting key fad6d80e85bbe624c21229b08ca40b9d
Jun 20 20:18:53 dist-test-slave-2hpg krb5kdc[29128](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986733, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.217.1@KRBTEST.COM for kudu/127.16.217.62@KRBTEST.COM
I20260620 20:18:53.571385 29819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:53.532474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.217.1:57923 (local address 127.16.217.62:40353)
0620 20:18:53.550181 (+ 17707us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:53.550189 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:53.550217 (+ 28us) server_negotiation.cc:407] Connection header received
0620 20:18:53.552004 (+ 1787us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:53.552009 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:53.552112 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:53.552202 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:18:53.555416 (+ 3214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:53.556480 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:53.558695 (+ 2215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:53.559257 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:53.559341 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:53.563796 (+ 4455us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:18:53.563805 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:18:53.563811 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:18:53.563858 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:18:53.567436 (+ 3578us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:53.568360 (+ 924us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:53.568366 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:53.568370 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:53.568520 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:53.569133 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:53.569138 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:53.569140 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:53.569507 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:18:53.569631 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0620 20:18:53.570747 (+ 1116us) server_negotiation.cc:299] Negotiation successful
0620 20:18:53.570993 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17554,"thread_start_us":16010,"threads_started":1}
WARNING: no policy specified for kudu/127.16.217.2@KRBTEST.COM; defaulting to no policy
I20260620 20:18:53.573745 29820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:53.533454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:40353 (local address 127.16.217.1:57923)
0620 20:18:53.545830 (+ 12376us) negotiation.cc:107] Waiting for socket to connect
0620 20:18:53.545908 (+ 78us) client_negotiation.cc:174] Beginning negotiation
0620 20:18:53.551565 (+ 5657us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:18:53.553812 (+ 2247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:18:53.553825 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:18:53.554434 (+ 609us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:18:53.555223 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:53.555246 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:53.556674 (+ 1428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:53.556679 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:53.558543 (+ 1864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:53.558556 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:53.559659 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:53.559665 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:53.559930 (+ 265us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:53.560712 (+ 782us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:18:53.560738 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:18:53.563537 (+ 2799us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:18:53.567603 (+ 4066us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:18:53.567613 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:18:53.567631 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:18:53.568222 (+ 591us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:18:53.568661 (+ 439us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:18:53.568665 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:18:53.568669 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:18:53.568999 (+ 330us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:18:53.569720 (+ 721us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:18:53.569730 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:18:53.570318 (+ 588us) client_negotiation.cc:769] Sending connection context
0620 20:18:53.571928 (+ 1610us) client_negotiation.cc:240] Negotiation successful
0620 20:18:53.572464 (+ 536us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12027,"thread_start_us":13735,"threads_started":1}
I20260620 20:18:53.576011 29817 heartbeater.cc:347] Connected to a master server at 127.16.217.62:40353
I20260620 20:18:53.576330 29817 heartbeater.cc:464] Registering TS with master...
I20260620 20:18:53.577152 29817 heartbeater.cc:511] Master 127.16.217.62:40353 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.217.2@KRBTEST.COM" created.
I20260620 20:18:53.579559 29604 ts_manager.cc:194] Registered new tserver with Master: 890d1d3512c14e0288ac4e0ce29ce66d (127.16.217.1:38219)
I20260620 20:18:53.581439 29604 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.217.1@KRBTEST.COM'} at 127.16.217.1:57923
Jun 20 20:18:53 dist-test-slave-2hpg krb5kdc[29128](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986733, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.217.62@KRBTEST.COM for HTTP/127.16.217.20@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.217.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.2.keytab.
Entry for principal kudu/127.16.217.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.2.keytab.
I20260620 20:18:53.632011 17252 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
/tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986536687152-17252-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986536687152-17252-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986536687152-17252-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986536687152-17252-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.16.217.20:49279/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.217.2:0
--local_ip_for_outbound_sockets=127.16.217.2
--webserver_interface=127.16.217.2
--webserver_port=0
--tserver_master_addrs=127.16.217.62:40353
--keytab_file=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.2.keytab
--principal=kudu/127.16.217.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.217.20:35163
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986536687152-17252-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986536687152-17252-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986536687152-17252-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 20:18:53 dist-test-slave-2hpg krb5kdc[29128](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986733, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.217.62@KRBTEST.COM for HTTP/127.16.217.20@KRBTEST.COM
W20260620 20:18:54.097982 29823 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:18:54.098417 29823 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:18:54.098588 29823 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:18:54.110040 29823 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:18:54.110301 29823 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:18:54.110553 29823 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.217.2
I20260620 20:18:54.126262 29823 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.217.20:35163
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986536687152-17252-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986536687152-17252-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.16.217.20:49279/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.2.keytab
--principal=kudu/127.16.217.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.217.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986536687152-17252-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.217.2
--webserver_port=0
--tserver_master_addrs=127.16.217.62:40353
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.217.2
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986536687152-17252-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:18:54.128319 29823 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:18:54.130235 29823 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:18:54.166848 29831 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:18:54.167490 29830 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:18:54.183583 29833 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:18:54.184384 29823 server_base.cc:1034] running on GCE node
I20260620 20:18:54.185420 29823 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:18:54.187755 29823 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:18:54.189126 29823 hybrid_clock.cc:648] HybridClock initialized: now 1781986734188984 us; error 104 us; skew 500 ppm
I20260620 20:18:54.189572 29823 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:18:54 dist-test-slave-2hpg krb5kdc[29128](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986734, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.217.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:18:54.193917 29823 init.cc:377] Logged in from keytab as kudu/127.16.217.2@KRBTEST.COM (short username kudu)
I20260620 20:18:54.195866 29823 webserver.cc:460] Webserver started at http://127.16.217.2:41231/ using document root <none> and password file <none>
I20260620 20:18:54.196756 29823 fs_manager.cc:359] Metadata directory not provided
I20260620 20:18:54.196889 29823 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:18:54.197252 29823 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 20:18:54 dist-test-slave-2hpg krb5kdc[29128](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986734, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.217.2@KRBTEST.COM for HTTP/127.16.217.20@KRBTEST.COM
I20260620 20:18:54.215874 29823 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986536687152-17252-0/minicluster-data/ts-1/data/instance:
uuid: "42865a9662cc44028f95ae95ee959729"
format_stamp: "Formatted at 2026-06-20 20:18:54 on dist-test-slave-2hpg"
server_key: "80557346d1d64ee423de758b2d256258"
server_key_iv: "fd0492685c1a792949080e17b613ceda"
server_key_version: "kuduclusterkey@0"
I20260620 20:18:54.228294 29823 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986536687152-17252-0/minicluster-data/ts-1/wal/instance:
uuid: "42865a9662cc44028f95ae95ee959729"
format_stamp: "Formatted at 2026-06-20 20:18:54 on dist-test-slave-2hpg"
server_key: "80557346d1d64ee423de758b2d256258"
server_key_iv: "fd0492685c1a792949080e17b613ceda"
server_key_version: "kuduclusterkey@0"
I20260620 20:18:54.244560 29823 fs_manager.cc:691] Time spent creating directory manager: real 0.016s user 0.005s sys 0.002s
I20260620 20:18:54.264317 29847 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:18:54.265996 29823 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260620 20:18:54.266170 29823 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986536687152-17252-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986536687152-17252-0/minicluster-data/ts-1/wal
uuid: "42865a9662cc44028f95ae95ee959729"
format_stamp: "Formatted at 2026-06-20 20:18:54 on dist-test-slave-2hpg"
server_key: "80557346d1d64ee423de758b2d256258"
server_key_iv: "fd0492685c1a792949080e17b613ceda"
server_key_version: "kuduclusterkey@0"
I20260620 20:18:54.266386 29823 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986536687152-17252-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986536687152-17252-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986536687152-17252-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:18:54.286145 29823 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:18:54.290452 29823 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:18:54.290776 29823 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:18:54.291889 29823 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:18:54.293437 29823 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:18:54.293512 29823 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:18:54.293653 29823 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:18:54.293709 29823 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:18:54.329977 29823 rpc_server.cc:307] RPC server started. Bound to: 127.16.217.2:38689
I20260620 20:18:54.329993 29959 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.217.2:38689 every 8 connection(s)
I20260620 20:18:54.331368 29823 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986536687152-17252-0/minicluster-data/ts-1/data/info.pb
I20260620 20:18:54.332795 17252 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu as pid 29823
I20260620 20:18:54.332958 17252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986536687152-17252-0/minicluster-data/ts-1/wal/instance
I20260620 20:18:54.333539 17252 external_mini_cluster.cc:1428] Setting key aa7f596cfbfc64ce09f45fa1070f4872
Jun 20 20:18:54 dist-test-slave-2hpg krb5kdc[29128](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986734, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.217.2@KRBTEST.COM for kudu/127.16.217.62@KRBTEST.COM
I20260620 20:18:54.356729 29962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:54.334459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.217.2:41453 (local address 127.16.217.62:40353)
0620 20:18:54.334953 (+ 494us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:54.334957 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:54.341617 (+ 6660us) server_negotiation.cc:407] Connection header received
0620 20:18:54.341707 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:54.341712 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:54.341799 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:54.341944 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:18:54.343871 (+ 1927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:54.344622 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:54.346492 (+ 1870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:54.347012 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:54.347157 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:54.350637 (+ 3480us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:18:54.350645 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:18:54.350651 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:18:54.350716 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:18:54.353489 (+ 2773us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:54.354322 (+ 833us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:54.354328 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:54.354331 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:54.354456 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:54.354886 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:54.354891 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:54.354895 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:54.355176 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:18:54.355329 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0620 20:18:54.356113 (+ 784us) server_negotiation.cc:299] Negotiation successful
0620 20:18:54.356333 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":354,"thread_start_us":246,"threads_started":1}
I20260620 20:18:54.357782 29963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:54.335198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:40353 (local address 127.16.217.2:41453)
0620 20:18:54.338502 (+ 3304us) negotiation.cc:107] Waiting for socket to connect
0620 20:18:54.338574 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0620 20:18:54.341279 (+ 2705us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:18:54.342229 (+ 950us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:18:54.342244 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:18:54.342891 (+ 647us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:18:54.343685 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:54.343704 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:54.344777 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:54.344782 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:54.346292 (+ 1510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:54.346305 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:54.347246 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:54.347253 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:54.347495 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:54.348197 (+ 702us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:18:54.348220 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:18:54.350408 (+ 2188us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:18:54.353645 (+ 3237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:18:54.353654 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:18:54.353672 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:18:54.354198 (+ 526us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:18:54.354576 (+ 378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:18:54.354581 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:18:54.354585 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:18:54.354766 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:18:54.355320 (+ 554us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:18:54.355328 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:18:54.355696 (+ 368us) client_negotiation.cc:769] Sending connection context
0620 20:18:54.356003 (+ 307us) client_negotiation.cc:240] Negotiation successful
0620 20:18:54.356534 (+ 531us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2972,"thread_start_us":2768,"threads_started":1}
I20260620 20:18:54.359319 29960 heartbeater.cc:347] Connected to a master server at 127.16.217.62:40353
I20260620 20:18:54.359648 29960 heartbeater.cc:464] Registering TS with master...
I20260620 20:18:54.360350 29960 heartbeater.cc:511] Master 127.16.217.62:40353 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.16.217.3@KRBTEST.COM; defaulting to no policy
I20260620 20:18:54.362048 29604 ts_manager.cc:194] Registered new tserver with Master: 42865a9662cc44028f95ae95ee959729 (127.16.217.2:38689)
I20260620 20:18:54.362944 29604 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.217.2@KRBTEST.COM'} at 127.16.217.2:41453
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.217.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.217.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.3.keytab.
Entry for principal kudu/127.16.217.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.3.keytab.
I20260620 20:18:54.398536 17252 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
/tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986536687152-17252-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986536687152-17252-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986536687152-17252-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986536687152-17252-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.16.217.20:49279/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.217.3:0
--local_ip_for_outbound_sockets=127.16.217.3
--webserver_interface=127.16.217.3
--webserver_port=0
--tserver_master_addrs=127.16.217.62:40353
--keytab_file=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.3.keytab
--principal=kudu/127.16.217.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.217.20:35163
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986536687152-17252-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986536687152-17252-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986536687152-17252-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 20:18:54.588723 29817 heartbeater.cc:503] Master 127.16.217.62:40353 was elected leader, sending a full tablet report...
W20260620 20:18:54.733551 29966 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:18:54.733923 29966 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:18:54.734032 29966 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:18:54.741317 29966 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:18:54.741425 29966 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:18:54.741544 29966 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.217.3
I20260620 20:18:54.751664 29966 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.217.20:35163
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986536687152-17252-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986536687152-17252-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.16.217.20:49279/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.3.keytab
--principal=kudu/127.16.217.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.217.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986536687152-17252-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.217.3
--webserver_port=0
--tserver_master_addrs=127.16.217.62:40353
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.217.3
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986536687152-17252-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:18:54.752910 29966 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:18:54.754202 29966 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:18:54.767330 29971 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:18:54.767357 29974 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:18:54.767417 29972 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:18:54.768158 29966 server_base.cc:1034] running on GCE node
I20260620 20:18:54.768806 29966 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:18:54.769829 29966 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:18:54.771272 29966 hybrid_clock.cc:648] HybridClock initialized: now 1781986734771203 us; error 63 us; skew 500 ppm
I20260620 20:18:54.771597 29966 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:18:54 dist-test-slave-2hpg krb5kdc[29128](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986734, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.217.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:18:54.774868 29966 init.cc:377] Logged in from keytab as kudu/127.16.217.3@KRBTEST.COM (short username kudu)
I20260620 20:18:54.776392 29966 webserver.cc:460] Webserver started at http://127.16.217.3:44451/ using document root <none> and password file <none>
I20260620 20:18:54.777102 29966 fs_manager.cc:359] Metadata directory not provided
I20260620 20:18:54.777245 29966 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:18:54.777540 29966 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 20:18:54 dist-test-slave-2hpg krb5kdc[29128](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986734, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.217.3@KRBTEST.COM for HTTP/127.16.217.20@KRBTEST.COM
I20260620 20:18:54.794983 29966 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986536687152-17252-0/minicluster-data/ts-2/data/instance:
uuid: "e3dc149d970842be80b46d8220cdf262"
format_stamp: "Formatted at 2026-06-20 20:18:54 on dist-test-slave-2hpg"
server_key: "caeb4639a41b1f568652a71197950598"
server_key_iv: "4ce8a4ed1ce985a5046d7b2be0be1c85"
server_key_version: "kuduclusterkey@0"
I20260620 20:18:54.795879 29966 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986536687152-17252-0/minicluster-data/ts-2/wal/instance:
uuid: "e3dc149d970842be80b46d8220cdf262"
format_stamp: "Formatted at 2026-06-20 20:18:54 on dist-test-slave-2hpg"
server_key: "caeb4639a41b1f568652a71197950598"
server_key_iv: "4ce8a4ed1ce985a5046d7b2be0be1c85"
server_key_version: "kuduclusterkey@0"
I20260620 20:18:54.801698 29966 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260620 20:18:54.819666 29981 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:18:54.821413 29966 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260620 20:18:54.821627 29966 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986536687152-17252-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986536687152-17252-0/minicluster-data/ts-2/wal
uuid: "e3dc149d970842be80b46d8220cdf262"
format_stamp: "Formatted at 2026-06-20 20:18:54 on dist-test-slave-2hpg"
server_key: "caeb4639a41b1f568652a71197950598"
server_key_iv: "4ce8a4ed1ce985a5046d7b2be0be1c85"
server_key_version: "kuduclusterkey@0"
I20260620 20:18:54.821931 29966 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986536687152-17252-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986536687152-17252-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986536687152-17252-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:18:54.853703 29966 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:18:54.858004 29966 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:18:54.858316 29966 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:18:54.859470 29966 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:18:54.860906 29966 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:18:54.861013 29966 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:18:54.861168 29966 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:18:54.861243 29966 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:18:54.896486 29966 rpc_server.cc:307] RPC server started. Bound to: 127.16.217.3:41719
I20260620 20:18:54.896521 30093 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.217.3:41719 every 8 connection(s)
I20260620 20:18:54.897950 29966 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986536687152-17252-0/minicluster-data/ts-2/data/info.pb
I20260620 20:18:54.902602 17252 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu as pid 29966
I20260620 20:18:54.902791 17252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986536687152-17252-0/minicluster-data/ts-2/wal/instance
I20260620 20:18:54.903304 17252 external_mini_cluster.cc:1428] Setting key e0c16c138e31357cac788d3bbdbf2fb2
Jun 20 20:18:54 dist-test-slave-2hpg krb5kdc[29128](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986734, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.217.3@KRBTEST.COM for kudu/127.16.217.62@KRBTEST.COM
I20260620 20:18:54.918217 30096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:54.900575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.217.3:53049 (local address 127.16.217.62:40353)
0620 20:18:54.901050 (+ 475us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:54.901054 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:54.902214 (+ 1160us) server_negotiation.cc:407] Connection header received
0620 20:18:54.904403 (+ 2189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:54.904408 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:54.904475 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:54.904559 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:18:54.906174 (+ 1615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:54.907049 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:54.908627 (+ 1578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:54.909067 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:54.909172 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:54.912394 (+ 3222us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:18:54.912400 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:18:54.912403 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:18:54.912434 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:18:54.915049 (+ 2615us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:54.915775 (+ 726us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:54.915779 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:54.915781 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:54.915910 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:54.916322 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:54.916325 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:54.916327 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:54.916604 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:18:54.916722 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0620 20:18:54.917616 (+ 894us) server_negotiation.cc:299] Negotiation successful
0620 20:18:54.917934 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":364,"thread_start_us":224,"threads_started":1}
I20260620 20:18:54.919387 30097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:54.901210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:40353 (local address 127.16.217.3:53049)
0620 20:18:54.902028 (+ 818us) negotiation.cc:107] Waiting for socket to connect
0620 20:18:54.902073 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 20:18:54.904042 (+ 1969us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:18:54.904833 (+ 791us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:18:54.904841 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:18:54.905291 (+ 450us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:18:54.906021 (+ 730us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:54.906038 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:54.907204 (+ 1166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:54.907208 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:54.908454 (+ 1246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:54.908465 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:54.909277 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:54.909282 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:54.909482 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:54.910190 (+ 708us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:18:54.910211 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:18:54.912195 (+ 1984us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:18:54.915200 (+ 3005us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:18:54.915207 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:18:54.915227 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:18:54.915660 (+ 433us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:18:54.915997 (+ 337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:18:54.916000 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:18:54.916003 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:18:54.916225 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:18:54.916722 (+ 497us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:18:54.916727 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:18:54.917183 (+ 456us) client_negotiation.cc:769] Sending connection context
0620 20:18:54.917489 (+ 306us) client_negotiation.cc:240] Negotiation successful
0620 20:18:54.918046 (+ 557us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":534,"thread_start_us":289,"threads_started":1}
I20260620 20:18:54.920786 30094 heartbeater.cc:347] Connected to a master server at 127.16.217.62:40353
I20260620 20:18:54.921116 30094 heartbeater.cc:464] Registering TS with master...
I20260620 20:18:54.921768 30094 heartbeater.cc:511] Master 127.16.217.62:40353 requested a full tablet report, sending...
I20260620 20:18:54.923162 29604 ts_manager.cc:194] Registered new tserver with Master: e3dc149d970842be80b46d8220cdf262 (127.16.217.3:41719)
I20260620 20:18:54.923979 29604 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.217.3@KRBTEST.COM'} at 127.16.217.3:53049
I20260620 20:18:54.931798 17252 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 20:18:54.933446 17252 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu pid 29684
Jun 20 20:18:54 dist-test-slave-2hpg krb5kdc[29128](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986731, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.217.1@KRBTEST.COM
I20260620 20:18:54.950413 30099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:54.934505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43950 (local address 127.16.217.1:38219)
0620 20:18:54.935173 (+ 668us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:54.935180 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:54.935206 (+ 26us) server_negotiation.cc:407] Connection header received
0620 20:18:54.935262 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:54.935269 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:54.935481 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:54.935612 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:18:54.937205 (+ 1593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:54.938307 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:54.939744 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:54.940308 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:54.940404 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:54.944931 (+ 4527us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:18:54.944945 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:18:54.944960 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:18:54.944998 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:18:54.947859 (+ 2861us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:54.948512 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:54.948520 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:54.948522 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:54.948677 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:54.948981 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:54.948984 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:54.948986 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:54.949301 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:18:54.949573 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0620 20:18:54.949760 (+ 187us) server_negotiation.cc:299] Negotiation successful
0620 20:18:54.950025 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":511,"thread_start_us":342,"threads_started":1}
I20260620 20:18:54.951728 29812 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:18:55.034480 17252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu with pid 29684
I20260620 20:18:55.050261 17252 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu pid 29823
Jun 20 20:18:55 dist-test-slave-2hpg krb5kdc[29128](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986731, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.217.2@KRBTEST.COM
I20260620 20:18:55.066915 30101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:55.051387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60224 (local address 127.16.217.2:38689)
0620 20:18:55.051954 (+ 567us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:55.051961 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:55.051987 (+ 26us) server_negotiation.cc:407] Connection header received
0620 20:18:55.052045 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:55.052051 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:55.052273 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:55.052405 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:18:55.053644 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:55.054698 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:55.056108 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:55.056685 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:55.056786 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:55.061266 (+ 4480us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:18:55.061278 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:18:55.061305 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:18:55.061339 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:18:55.064274 (+ 2935us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:55.064901 (+ 627us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:55.064908 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:55.064910 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:55.065030 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:55.065335 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:55.065338 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:55.065340 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:55.065678 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:18:55.065987 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0620 20:18:55.066263 (+ 276us) server_negotiation.cc:299] Negotiation successful
0620 20:18:55.066472 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":430,"thread_start_us":318,"threads_started":1}
I20260620 20:18:55.068233 29955 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:18:55.150635 17252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu with pid 29823
I20260620 20:18:55.166445 17252 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu pid 29966
Jun 20 20:18:55 dist-test-slave-2hpg krb5kdc[29128](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986731, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.217.3@KRBTEST.COM
I20260620 20:18:55.183145 30103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:55.167419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38544 (local address 127.16.217.3:41719)
0620 20:18:55.167958 (+ 539us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:55.167965 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:55.167994 (+ 29us) server_negotiation.cc:407] Connection header received
0620 20:18:55.168068 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:55.168074 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:55.168283 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:55.168414 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:18:55.169677 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:55.170848 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:55.172075 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:55.172621 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:55.172726 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:55.177480 (+ 4754us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:18:55.177491 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:18:55.177506 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:18:55.177542 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:18:55.180507 (+ 2965us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:55.181138 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:55.181145 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:55.181147 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:55.181287 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:55.181680 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:55.181684 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:55.181686 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:55.182131 (+ 445us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:18:55.182386 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0620 20:18:55.182655 (+ 269us) server_negotiation.cc:299] Negotiation successful
0620 20:18:55.182858 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":406,"thread_start_us":254,"threads_started":1}
I20260620 20:18:55.184475 30089 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:18:55.272825 17252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu with pid 29966
I20260620 20:18:55.289288 17252 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu pid 29572
I20260620 20:18:55.290568 29634 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:18:55.397120 17252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu with pid 29572
I20260620 20:18:55.418998 17252 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
/tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986536687152-17252-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986536687152-17252-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986536687152-17252-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986536687152-17252-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.217.62:40353
--webserver_interface=127.16.217.62
--webserver_port=46209
--keytab_file=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.62.keytab
--principal=kudu/127.16.217.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.217.20:35163
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.217.62:40353
--ranger_config_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986536687152-17252-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.16.217.20:49279/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.16.217.20:49279/kms with env {KRB5CCNAME=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986536687152-17252-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986536687152-17252-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986536687152-17252-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:18:55.769582 30106 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:18:55.769995 30106 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:18:55.770165 30106 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:18:55.777490 30106 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:18:55.777665 30106 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:18:55.777776 30106 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:18:55.777907 30106 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:18:55.777984 30106 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:18:55.788182 30106 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.217.20:35163
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986536687152-17252-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986536687152-17252-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.16.217.20:49279/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.16.217.62:40353
--ranger_config_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986536687152-17252-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.62.keytab
--principal=kudu/127.16.217.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.217.62:40353
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986536687152-17252-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.217.62
--webserver_port=46209
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986536687152-17252-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:18:55.789587 30106 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:18:55.790977 30106 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:18:55.803532 30111 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:18:55.803480 30112 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:18:55.803426 30114 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:18:55.804231 30106 server_base.cc:1034] running on GCE node
I20260620 20:18:55.804999 30106 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:18:55.806506 30106 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:18:55.807758 30106 hybrid_clock.cc:648] HybridClock initialized: now 1781986735807722 us; error 56 us; skew 500 ppm
I20260620 20:18:55.808138 30106 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:18:55 dist-test-slave-2hpg krb5kdc[29128](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986735, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.217.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:18:55.811482 30106 init.cc:377] Logged in from keytab as kudu/127.16.217.62@KRBTEST.COM (short username kudu)
I20260620 20:18:55.813068 30106 webserver.cc:460] Webserver started at http://127.16.217.62:46209/ using document root <none> and password file <none>
I20260620 20:18:55.813782 30106 fs_manager.cc:359] Metadata directory not provided
I20260620 20:18:55.813961 30106 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 20:18:55 dist-test-slave-2hpg krb5kdc[29128](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986735, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.217.62@KRBTEST.COM for HTTP/127.16.217.20@KRBTEST.COM
I20260620 20:18:55.868296 30106 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.001s sys 0.004s
I20260620 20:18:55.871631 30122 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:18:55.873361 30106 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 20:18:55.873629 30106 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986536687152-17252-0/minicluster-data/master-0/data,/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986536687152-17252-0/minicluster-data/master-0/wal
uuid: "12eb0dcc6c7f4782be00f8df729c5aba"
format_stamp: "Formatted at 2026-06-20 20:18:51 on dist-test-slave-2hpg"
server_key: "c88d4877ff2d8a3b71b8addddcd83823"
server_key_iv: "ab29dc8ff0c7f6b4840a5b299c75f748"
server_key_version: "kuduclusterkey@0"
I20260620 20:18:55.874858 30106 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986536687152-17252-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986536687152-17252-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986536687152-17252-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:18:55.890592 30106 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:18:55.895260 30106 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:18:55.895612 30106 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:18:55.916257 30106 rpc_server.cc:307] RPC server started. Bound to: 127.16.217.62:40353
I20260620 20:18:55.916275 30173 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.217.62:40353 every 8 connection(s)
I20260620 20:18:55.918059 30106 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986536687152-17252-0/minicluster-data/master-0/data/info.pb
I20260620 20:18:55.923331 17252 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu as pid 30106
I20260620 20:18:55.924401 17252 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
/tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986536687152-17252-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986536687152-17252-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986536687152-17252-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986536687152-17252-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.16.217.20:49279/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.217.1:38219
--local_ip_for_outbound_sockets=127.16.217.1
--tserver_master_addrs=127.16.217.62:40353
--webserver_port=43495
--webserver_interface=127.16.217.1
--keytab_file=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.1.keytab
--principal=kudu/127.16.217.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.217.20:35163
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.16.217.20:49279/kms with env {KRB5CCNAME=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986536687152-17252-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986536687152-17252-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986536687152-17252-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 20:18:55.926373 30174 sys_catalog.cc:263] Verifying existing consensus state
I20260620 20:18:55.932005 30174 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 12eb0dcc6c7f4782be00f8df729c5aba: Bootstrap starting.
I20260620 20:18:55.951228 30174 log.cc:824] T 00000000000000000000000000000000 P 12eb0dcc6c7f4782be00f8df729c5aba: Log is configured to *not* fsync() on all Append() calls
I20260620 20:18:55.960425 30174 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 12eb0dcc6c7f4782be00f8df729c5aba: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 20:18:55.960975 30174 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 12eb0dcc6c7f4782be00f8df729c5aba: Bootstrap complete.
I20260620 20:18:55.970085 30174 raft_consensus.cc:348] T 00000000000000000000000000000000 P 12eb0dcc6c7f4782be00f8df729c5aba [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12eb0dcc6c7f4782be00f8df729c5aba" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 40353 } }
I20260620 20:18:55.970934 30174 raft_consensus.cc:729] T 00000000000000000000000000000000 P 12eb0dcc6c7f4782be00f8df729c5aba [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12eb0dcc6c7f4782be00f8df729c5aba, State: Initialized, Role: FOLLOWER
I20260620 20:18:55.971675 30174 consensus_queue.cc:260] T 00000000000000000000000000000000 P 12eb0dcc6c7f4782be00f8df729c5aba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12eb0dcc6c7f4782be00f8df729c5aba" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 40353 } }
I20260620 20:18:55.971906 30174 raft_consensus.cc:388] T 00000000000000000000000000000000 P 12eb0dcc6c7f4782be00f8df729c5aba [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 20:18:55.972020 30174 raft_consensus.cc:482] T 00000000000000000000000000000000 P 12eb0dcc6c7f4782be00f8df729c5aba [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 20:18:55.972222 30174 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 12eb0dcc6c7f4782be00f8df729c5aba [term 1 FOLLOWER]: Advancing to term 2
I20260620 20:18:55.976174 30174 raft_consensus.cc:504] T 00000000000000000000000000000000 P 12eb0dcc6c7f4782be00f8df729c5aba [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12eb0dcc6c7f4782be00f8df729c5aba" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 40353 } }
I20260620 20:18:55.976692 30174 leader_election.cc:302] T 00000000000000000000000000000000 P 12eb0dcc6c7f4782be00f8df729c5aba [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 12eb0dcc6c7f4782be00f8df729c5aba; no voters:
I20260620 20:18:55.977329 30174 leader_election.cc:288] T 00000000000000000000000000000000 P 12eb0dcc6c7f4782be00f8df729c5aba [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 20:18:55.977465 30178 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 12eb0dcc6c7f4782be00f8df729c5aba [term 2 FOLLOWER]: Leader election won for term 2
I20260620 20:18:55.978678 30178 raft_consensus.cc:686] T 00000000000000000000000000000000 P 12eb0dcc6c7f4782be00f8df729c5aba [term 2 LEADER]: Becoming Leader. State: Replica: 12eb0dcc6c7f4782be00f8df729c5aba, State: Running, Role: LEADER
I20260620 20:18:55.979264 30178 consensus_queue.cc:237] T 00000000000000000000000000000000 P 12eb0dcc6c7f4782be00f8df729c5aba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12eb0dcc6c7f4782be00f8df729c5aba" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 40353 } }
I20260620 20:18:55.979792 30174 sys_catalog.cc:564] T 00000000000000000000000000000000 P 12eb0dcc6c7f4782be00f8df729c5aba [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:18:55.982015 30179 sys_catalog.cc:455] T 00000000000000000000000000000000 P 12eb0dcc6c7f4782be00f8df729c5aba [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "12eb0dcc6c7f4782be00f8df729c5aba" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12eb0dcc6c7f4782be00f8df729c5aba" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 40353 } } }
I20260620 20:18:55.982460 30179 sys_catalog.cc:458] T 00000000000000000000000000000000 P 12eb0dcc6c7f4782be00f8df729c5aba [sys.catalog]: This master's current role is: LEADER
I20260620 20:18:55.987100 30180 sys_catalog.cc:455] T 00000000000000000000000000000000 P 12eb0dcc6c7f4782be00f8df729c5aba [sys.catalog]: SysCatalogTable state changed. Reason: New leader 12eb0dcc6c7f4782be00f8df729c5aba. Latest consensus state: current_term: 2 leader_uuid: "12eb0dcc6c7f4782be00f8df729c5aba" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12eb0dcc6c7f4782be00f8df729c5aba" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 40353 } } }
I20260620 20:18:55.987473 30180 sys_catalog.cc:458] T 00000000000000000000000000000000 P 12eb0dcc6c7f4782be00f8df729c5aba [sys.catalog]: This master's current role is: LEADER
I20260620 20:18:55.990497 30184 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 20:18:55.997665 30184 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 20:18:55.999255 30184 catalog_manager.cc:1259] Loaded cluster ID: 41e1c0e688564eed8e9496071fd3729e
I20260620 20:18:55.999460 30184 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 20:18:56.003827 30184 catalog_manager.cc:1504] Loading token signing keys...
I20260620 20:18:56.005319 30184 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 12eb0dcc6c7f4782be00f8df729c5aba: Loaded TSK: 0
I20260620 20:18:56.007220 30184 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 20:18:56.010996 30174 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986536687152-17252-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
W20260620 20:18:56.389886 30176 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:18:56.390331 30176 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:18:56.390532 30176 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:18:56.402275 30176 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:18:56.402503 30176 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:18:56.402740 30176 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.217.1
I20260620 20:18:56.416927 30176 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.217.20:35163
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986536687152-17252-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986536687152-17252-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.16.217.20:49279/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.1.keytab
--principal=kudu/127.16.217.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.217.1:38219
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986536687152-17252-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.217.1
--webserver_port=43495
--tserver_master_addrs=127.16.217.62:40353
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.217.1
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986536687152-17252-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:18:56.418895 30176 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:18:56.420781 30176 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:18:56.457130 30212 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:18:56.460285 30214 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:18:56.461045 30211 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:18:56.463442 30176 server_base.cc:1034] running on GCE node
I20260620 20:18:56.464540 30176 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:18:56.473824 30176 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:18:56.477419 30176 hybrid_clock.cc:648] HybridClock initialized: now 1781986736477271 us; error 137 us; skew 500 ppm
I20260620 20:18:56.478044 30176 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:18:56 dist-test-slave-2hpg krb5kdc[29128](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986736, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.217.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:18:56.488976 30176 init.cc:377] Logged in from keytab as kudu/127.16.217.1@KRBTEST.COM (short username kudu)
I20260620 20:18:56.503352 30176 webserver.cc:460] Webserver started at http://127.16.217.1:43495/ using document root <none> and password file <none>
I20260620 20:18:56.504247 30176 fs_manager.cc:359] Metadata directory not provided
I20260620 20:18:56.504470 30176 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 20:18:56 dist-test-slave-2hpg krb5kdc[29128](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986736, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.217.1@KRBTEST.COM for HTTP/127.16.217.20@KRBTEST.COM
I20260620 20:18:56.542613 30176 fs_manager.cc:709] Time spent opening directory manager: real 0.008s user 0.002s sys 0.001s
I20260620 20:18:56.562649 30222 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:18:56.564908 30176 fs_manager.cc:725] Time spent opening block manager: real 0.020s user 0.009s sys 0.011s
I20260620 20:18:56.565114 30176 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986536687152-17252-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986536687152-17252-0/minicluster-data/ts-0/wal
uuid: "890d1d3512c14e0288ac4e0ce29ce66d"
format_stamp: "Formatted at 2026-06-20 20:18:53 on dist-test-slave-2hpg"
server_key: "d0fcf224af91cc0ee838039aa68e21b7"
server_key_iv: "1edf6d5edcf0a07d33b1f794585d6fd5"
server_key_version: "kuduclusterkey@0"
I20260620 20:18:56.566677 30176 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986536687152-17252-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986536687152-17252-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986536687152-17252-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:18:56.612131 30176 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:18:56.617466 30176 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:18:56.617797 30176 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:18:56.620033 30176 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:18:56.625021 30176 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:18:56.625176 30176 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:18:56.625326 30176 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:18:56.625408 30176 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:18:56.670149 30176 rpc_server.cc:307] RPC server started. Bound to: 127.16.217.1:38219
I20260620 20:18:56.670184 30335 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.217.1:38219 every 8 connection(s)
I20260620 20:18:56.672204 30176 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986536687152-17252-0/minicluster-data/ts-0/data/info.pb
I20260620 20:18:56.673372 17252 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu as pid 30176
I20260620 20:18:56.674255 17252 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
/tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986536687152-17252-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986536687152-17252-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986536687152-17252-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986536687152-17252-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.16.217.20:49279/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.217.2:38689
--local_ip_for_outbound_sockets=127.16.217.2
--tserver_master_addrs=127.16.217.62:40353
--webserver_port=41231
--webserver_interface=127.16.217.2
--keytab_file=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.2.keytab
--principal=kudu/127.16.217.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.217.20:35163
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.16.217.20:49279/kms with env {KRB5CCNAME=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986536687152-17252-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986536687152-17252-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986536687152-17252-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 20:18:56 dist-test-slave-2hpg krb5kdc[29128](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986736, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.217.1@KRBTEST.COM for kudu/127.16.217.62@KRBTEST.COM
I20260620 20:18:56.721648 30338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:56.677528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:40353 (local address 127.16.217.1:56085)
0620 20:18:56.691998 (+ 14470us) negotiation.cc:107] Waiting for socket to connect
0620 20:18:56.692063 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0620 20:18:56.694470 (+ 2407us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:18:56.696603 (+ 2133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:18:56.696615 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:18:56.697268 (+ 653us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:18:56.699053 (+ 1785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:56.699081 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:56.701129 (+ 2048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:56.701134 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:56.702839 (+ 1705us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:56.702866 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:56.704813 (+ 1947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:56.704822 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:56.705104 (+ 282us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:56.708751 (+ 3647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:18:56.708788 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:18:56.711452 (+ 2664us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:18:56.716299 (+ 4847us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:18:56.716309 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:18:56.716329 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:18:56.716954 (+ 625us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:18:56.717470 (+ 516us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:18:56.717475 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:18:56.717479 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:18:56.717734 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:18:56.718896 (+ 1162us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:18:56.718905 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:18:56.719337 (+ 432us) client_negotiation.cc:769] Sending connection context
0620 20:18:56.719632 (+ 295us) client_negotiation.cc:240] Negotiation successful
0620 20:18:56.720218 (+ 586us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14151,"thread_start_us":13903,"threads_started":1}
I20260620 20:18:56.725028 30339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:56.678734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.217.1:56085 (local address 127.16.217.62:40353)
0620 20:18:56.691302 (+ 12568us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:56.691332 (+ 30us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:56.693991 (+ 2659us) server_negotiation.cc:407] Connection header received
0620 20:18:56.695402 (+ 1411us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:56.695423 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:56.695963 (+ 540us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:56.696416 (+ 453us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:18:56.699325 (+ 2909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:56.700940 (+ 1615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:56.703032 (+ 2092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:56.704138 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:56.704302 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:56.711852 (+ 7550us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:18:56.711865 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:18:56.711890 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:18:56.711949 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:18:56.716114 (+ 4165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:56.717135 (+ 1021us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:56.717146 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:56.717153 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:56.717347 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:56.717932 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:56.717938 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:56.717942 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:56.718618 (+ 676us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:18:56.718824 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0620 20:18:56.722603 (+ 3779us) server_negotiation.cc:299] Negotiation successful
0620 20:18:56.723115 (+ 512us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12255,"thread_start_us":12008,"threads_started":1}
I20260620 20:18:56.728833 30336 heartbeater.cc:347] Connected to a master server at 127.16.217.62:40353
I20260620 20:18:56.729177 30336 heartbeater.cc:464] Registering TS with master...
I20260620 20:18:56.730103 30336 heartbeater.cc:511] Master 127.16.217.62:40353 requested a full tablet report, sending...
W20260620 20:18:56.733560 30336 heartbeater.cc:644] Failed to heartbeat to 127.16.217.62:40353 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260620 20:18:56.733795 30336 heartbeater.cc:464] Registering TS with master...
I20260620 20:18:56.734123 30336 heartbeater.cc:511] Master 127.16.217.62:40353 requested a full tablet report, sending...
I20260620 20:18:56.735386 30336 heartbeater.cc:464] Registering TS with master...
I20260620 20:18:56.735658 30336 heartbeater.cc:511] Master 127.16.217.62:40353 requested a full tablet report, sending...
W20260620 20:18:56.736737 30336 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 20:18:57.103991 30340 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:18:57.104374 30340 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:18:57.104545 30340 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:18:57.112146 30340 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:18:57.112313 30340 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:18:57.112506 30340 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.217.2
I20260620 20:18:57.123570 30340 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.217.20:35163
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986536687152-17252-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986536687152-17252-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.16.217.20:49279/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.2.keytab
--principal=kudu/127.16.217.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.217.2:38689
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986536687152-17252-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.217.2
--webserver_port=41231
--tserver_master_addrs=127.16.217.62:40353
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.217.2
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986536687152-17252-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:18:57.125111 30340 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:18:57.126569 30340 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:18:57.142349 30357 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:18:57.142285 30355 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:18:57.142500 30354 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:18:57.146028 30340 server_base.cc:1034] running on GCE node
I20260620 20:18:57.146973 30340 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:18:57.148422 30340 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:18:57.155953 30340 hybrid_clock.cc:648] HybridClock initialized: now 1781986737155850 us; error 67 us; skew 500 ppm
I20260620 20:18:57.156422 30340 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:18:57 dist-test-slave-2hpg krb5kdc[29128](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986737, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.217.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:18:57.161064 30340 init.cc:377] Logged in from keytab as kudu/127.16.217.2@KRBTEST.COM (short username kudu)
I20260620 20:18:57.186105 30340 webserver.cc:460] Webserver started at http://127.16.217.2:41231/ using document root <none> and password file <none>
I20260620 20:18:57.187266 30340 fs_manager.cc:359] Metadata directory not provided
I20260620 20:18:57.187458 30340 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 20:18:57 dist-test-slave-2hpg krb5kdc[29128](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986737, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.217.2@KRBTEST.COM for HTTP/127.16.217.20@KRBTEST.COM
I20260620 20:18:57.231225 30340 fs_manager.cc:709] Time spent opening directory manager: real 0.015s user 0.009s sys 0.005s
I20260620 20:18:57.235503 30367 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:18:57.237505 30340 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260620 20:18:57.237844 30340 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986536687152-17252-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986536687152-17252-0/minicluster-data/ts-1/wal
uuid: "42865a9662cc44028f95ae95ee959729"
format_stamp: "Formatted at 2026-06-20 20:18:54 on dist-test-slave-2hpg"
server_key: "80557346d1d64ee423de758b2d256258"
server_key_iv: "fd0492685c1a792949080e17b613ceda"
server_key_version: "kuduclusterkey@0"
I20260620 20:18:57.239466 30340 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986536687152-17252-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986536687152-17252-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986536687152-17252-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:18:57.274803 30340 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:18:57.280478 30340 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:18:57.280877 30340 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:18:57.282266 30340 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:18:57.286828 30340 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:18:57.287017 30340 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260620 20:18:57.287184 30340 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:18:57.287318 30340 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 20:18:57 dist-test-slave-2hpg krb5kdc[29128](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781986737, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.217.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:18:57.355929 30340 rpc_server.cc:307] RPC server started. Bound to: 127.16.217.2:38689
I20260620 20:18:57.355978 30479 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.217.2:38689 every 8 connection(s)
I20260620 20:18:57.357365 30340 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986536687152-17252-0/minicluster-data/ts-1/data/info.pb
I20260620 20:18:57.362272 17252 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu as pid 30340
I20260620 20:18:57.363030 17252 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
/tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986536687152-17252-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986536687152-17252-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986536687152-17252-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986536687152-17252-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.16.217.20:49279/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.217.3:41719
--local_ip_for_outbound_sockets=127.16.217.3
--tserver_master_addrs=127.16.217.62:40353
--webserver_port=44451
--webserver_interface=127.16.217.3
--keytab_file=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.3.keytab
--principal=kudu/127.16.217.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.217.20:35163
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.16.217.20:49279/kms with env {KRB5CCNAME=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986536687152-17252-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986536687152-17252-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986536687152-17252-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 20:18:57 dist-test-slave-2hpg krb5kdc[29128](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986737, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.217.2@KRBTEST.COM for kudu/127.16.217.62@KRBTEST.COM
I20260620 20:18:57.401333 30482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:57.360729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.217.2:36769 (local address 127.16.217.62:40353)
0620 20:18:57.380412 (+ 19683us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:57.380420 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:57.380449 (+ 29us) server_negotiation.cc:407] Connection header received
0620 20:18:57.382466 (+ 2017us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:57.382472 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:57.382585 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:57.382710 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:18:57.384307 (+ 1597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:57.385063 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:57.386979 (+ 1916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:57.387636 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:57.387759 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:57.392720 (+ 4961us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:18:57.392727 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:18:57.392733 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:18:57.392775 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:18:57.396834 (+ 4059us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:57.397928 (+ 1094us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:57.397935 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:57.397939 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:57.398101 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:57.398674 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:57.398679 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:57.398683 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:57.399048 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:18:57.399216 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0620 20:18:57.400664 (+ 1448us) server_negotiation.cc:299] Negotiation successful
0620 20:18:57.400906 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19555,"thread_start_us":19409,"threads_started":1}
I20260620 20:18:57.403715 30483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:57.361445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:40353 (local address 127.16.217.2:36769)
0620 20:18:57.379587 (+ 18142us) negotiation.cc:107] Waiting for socket to connect
0620 20:18:57.379699 (+ 112us) client_negotiation.cc:174] Beginning negotiation
0620 20:18:57.382126 (+ 2427us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:18:57.382996 (+ 870us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:18:57.383006 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:18:57.383487 (+ 481us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:18:57.384111 (+ 624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:57.384129 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:57.385221 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:57.385226 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:57.386800 (+ 1574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:57.386815 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:57.388112 (+ 1297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:57.388120 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:57.388424 (+ 304us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:57.389326 (+ 902us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:18:57.389363 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:18:57.392316 (+ 2953us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:18:57.397035 (+ 4719us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:18:57.397046 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:18:57.397080 (+ 34us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:18:57.397737 (+ 657us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:18:57.398245 (+ 508us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:18:57.398250 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:18:57.398253 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:18:57.398535 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:18:57.399291 (+ 756us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:18:57.399300 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:18:57.399782 (+ 482us) client_negotiation.cc:769] Sending connection context
0620 20:18:57.400111 (+ 329us) client_negotiation.cc:240] Negotiation successful
0620 20:18:57.402248 (+ 2137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17771,"thread_start_us":17546,"threads_started":1}
I20260620 20:18:57.406188 30480 heartbeater.cc:347] Connected to a master server at 127.16.217.62:40353
I20260620 20:18:57.406553 30480 heartbeater.cc:464] Registering TS with master...
I20260620 20:18:57.407459 30480 heartbeater.cc:511] Master 127.16.217.62:40353 requested a full tablet report, sending...
I20260620 20:18:57.409685 30139 ts_manager.cc:194] Registered new tserver with Master: 42865a9662cc44028f95ae95ee959729 (127.16.217.2:38689)
I20260620 20:18:57.411891 30139 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.217.2@KRBTEST.COM'} at 127.16.217.2:36769
I20260620 20:18:57.736896 30336 heartbeater.cc:464] Registering TS with master...
I20260620 20:18:57.737315 30336 heartbeater.cc:511] Master 127.16.217.62:40353 requested a full tablet report, sending...
I20260620 20:18:57.738672 30139 ts_manager.cc:194] Registered new tserver with Master: 890d1d3512c14e0288ac4e0ce29ce66d (127.16.217.1:38219)
I20260620 20:18:57.739727 30139 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.217.1@KRBTEST.COM'} at 127.16.217.1:56085
Jun 20 20:18:57 dist-test-slave-2hpg krb5kdc[29128](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986737, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.217.62@KRBTEST.COM for HTTP/127.16.217.20@KRBTEST.COM
W20260620 20:18:57.876654 30485 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:18:57.877115 30485 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:18:57.877290 30485 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:18:57.889010 30485 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:18:57.889197 30485 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:18:57.889396 30485 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.217.3
I20260620 20:18:57.905431 30485 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.217.20:35163
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986536687152-17252-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986536687152-17252-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.16.217.20:49279/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.3.keytab
--principal=kudu/127.16.217.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.217.3:41719
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986536687152-17252-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.217.3
--webserver_port=44451
--tserver_master_addrs=127.16.217.62:40353
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.217.3
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986536687152-17252-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:18:57.907318 30485 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:18:57.909070 30485 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:18:57.958500 30491 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:18:57.959149 30490 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:18:57.961339 30494 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:18:57.961807 30485 server_base.cc:1034] running on GCE node
I20260620 20:18:57.962802 30485 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:18:57.969102 30485 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:18:57.970429 30485 hybrid_clock.cc:648] HybridClock initialized: now 1781986737970354 us; error 78 us; skew 500 ppm
I20260620 20:18:57.970937 30485 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:18:57 dist-test-slave-2hpg krb5kdc[29128](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986737, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.217.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:18:57.975440 30485 init.cc:377] Logged in from keytab as kudu/127.16.217.3@KRBTEST.COM (short username kudu)
I20260620 20:18:58.000916 30485 webserver.cc:460] Webserver started at http://127.16.217.3:44451/ using document root <none> and password file <none>
I20260620 20:18:58.002147 30485 fs_manager.cc:359] Metadata directory not provided
I20260620 20:18:58.002395 30485 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 20:18:58 dist-test-slave-2hpg krb5kdc[29128](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986737, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.217.3@KRBTEST.COM for HTTP/127.16.217.20@KRBTEST.COM
I20260620 20:18:58.042212 30485 fs_manager.cc:709] Time spent opening directory manager: real 0.012s user 0.011s sys 0.000s
I20260620 20:18:58.058561 30501 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:18:58.063452 30485 fs_manager.cc:725] Time spent opening block manager: real 0.019s user 0.009s sys 0.007s
I20260620 20:18:58.063742 30485 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986536687152-17252-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986536687152-17252-0/minicluster-data/ts-2/wal
uuid: "e3dc149d970842be80b46d8220cdf262"
format_stamp: "Formatted at 2026-06-20 20:18:54 on dist-test-slave-2hpg"
server_key: "caeb4639a41b1f568652a71197950598"
server_key_iv: "4ce8a4ed1ce985a5046d7b2be0be1c85"
server_key_version: "kuduclusterkey@0"
I20260620 20:18:58.065259 30485 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986536687152-17252-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986536687152-17252-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986536687152-17252-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:18:58.120541 30485 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:18:58.128176 30485 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:18:58.128537 30485 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:18:58.143141 30485 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:18:58.154832 30485 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:18:58.155041 30485 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260620 20:18:58.155222 30485 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:18:58.155336 30485 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 20:18:58 dist-test-slave-2hpg krb5kdc[29128](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986737, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.217.62@KRBTEST.COM for HTTP/127.16.217.20@KRBTEST.COM
I20260620 20:18:58.390942 30485 rpc_server.cc:307] RPC server started. Bound to: 127.16.217.3:41719
I20260620 20:18:58.391026 30614 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.217.3:41719 every 8 connection(s)
I20260620 20:18:58.392877 30485 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986536687152-17252-0/minicluster-data/ts-2/data/info.pb
I20260620 20:18:58.400877 17252 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu as pid 30485
I20260620 20:18:58.416888 30480 heartbeater.cc:503] Master 127.16.217.62:40353 was elected leader, sending a full tablet report...
Jun 20 20:18:58 dist-test-slave-2hpg krb5kdc[29128](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986737, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.217.3@KRBTEST.COM for kudu/127.16.217.62@KRBTEST.COM
I20260620 20:18:58.455322 30618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:58.401950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60184 (local address 127.16.217.62:40353)
0620 20:18:58.430819 (+ 28869us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:58.430826 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:58.430848 (+ 22us) server_negotiation.cc:407] Connection header received
0620 20:18:58.430911 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:58.430915 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:58.431017 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:58.431159 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:18:58.435121 (+ 3962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:58.436319 (+ 1198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:58.438849 (+ 2530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:58.439446 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:58.439570 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:58.446108 (+ 6538us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:18:58.446116 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:18:58.446121 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:18:58.446161 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:18:58.450515 (+ 4354us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:58.452672 (+ 2157us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:58.452679 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:58.452686 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:58.452853 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:58.453668 (+ 815us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:58.453676 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:58.453683 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:58.454061 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:18:58.454333 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0620 20:18:58.454676 (+ 343us) server_negotiation.cc:299] Negotiation successful
0620 20:18:58.454908 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28729,"thread_start_us":21824,"threads_started":1}
I20260620 20:18:58.455979 30619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:58.396943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.217.3:50119 (local address 127.16.217.62:40353)
0620 20:18:58.422929 (+ 25986us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:58.422939 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:58.433276 (+ 10337us) server_negotiation.cc:407] Connection header received
0620 20:18:58.434992 (+ 1716us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:58.434997 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:58.435081 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:58.435199 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:18:58.437600 (+ 2401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:58.438787 (+ 1187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:58.441462 (+ 2675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:58.442208 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:58.442297 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:58.446941 (+ 4644us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:18:58.446948 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:18:58.446952 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:18:58.447010 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:18:58.451287 (+ 4277us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:58.452101 (+ 814us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:58.452106 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:58.452110 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:58.452260 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:58.452861 (+ 601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:58.452866 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:58.452870 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:58.453264 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:18:58.453399 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0620 20:18:58.455467 (+ 2068us) server_negotiation.cc:299] Negotiation successful
0620 20:18:58.455685 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25800,"thread_start_us":34367,"threads_started":1}
I20260620 20:18:58.456923 30617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:58.396363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:40353 (local address 127.16.217.3:50119)
0620 20:18:58.432063 (+ 35700us) negotiation.cc:107] Waiting for socket to connect
0620 20:18:58.432145 (+ 82us) client_negotiation.cc:174] Beginning negotiation
0620 20:18:58.434647 (+ 2502us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:18:58.435705 (+ 1058us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:18:58.435720 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:18:58.436421 (+ 701us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:18:58.437326 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:58.437358 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:58.438990 (+ 1632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:58.438996 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:58.441194 (+ 2198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:58.441214 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:58.442641 (+ 1427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:58.442647 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:58.442945 (+ 298us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:58.443763 (+ 818us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:18:58.443792 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:18:58.446654 (+ 2862us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:18:58.451440 (+ 4786us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:18:58.451451 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:18:58.451472 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:18:58.451960 (+ 488us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:18:58.452407 (+ 447us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:18:58.452412 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:18:58.452416 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:18:58.452701 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:18:58.453508 (+ 807us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:18:58.453517 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:18:58.454051 (+ 534us) client_negotiation.cc:769] Sending connection context
0620 20:18:58.454402 (+ 351us) client_negotiation.cc:240] Negotiation successful
0620 20:18:58.454994 (+ 592us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35324,"thread_start_us":39959,"threads_started":1}
I20260620 20:18:58.458407 30615 heartbeater.cc:347] Connected to a master server at 127.16.217.62:40353
I20260620 20:18:58.458739 30615 heartbeater.cc:464] Registering TS with master...
I20260620 20:18:58.459452 30615 heartbeater.cc:511] Master 127.16.217.62:40353 requested a full tablet report, sending...
I20260620 20:18:58.460884 30139 ts_manager.cc:194] Registered new tserver with Master: e3dc149d970842be80b46d8220cdf262 (127.16.217.3:41719)
I20260620 20:18:58.462131 30139 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.217.3@KRBTEST.COM'} at 127.16.217.3:50119
I20260620 20:18:58.471459 17252 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 20:18:58.490945 30619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:58.475191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60198 (local address 127.16.217.62:40353)
0620 20:18:58.475407 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:58.475413 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:58.475929 (+ 516us) server_negotiation.cc:407] Connection header received
0620 20:18:58.476099 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:58.476105 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:58.476188 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:58.476297 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:18:58.478245 (+ 1948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:58.479449 (+ 1204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:58.481532 (+ 2083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:58.482088 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:58.482247 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:58.485557 (+ 3310us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:18:58.485562 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:18:58.485566 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:18:58.485601 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:18:58.488229 (+ 2628us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:58.489028 (+ 799us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:58.489031 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:58.489034 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:58.489154 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:58.489556 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:58.489559 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:58.489561 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:58.489823 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:18:58.490058 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0620 20:18:58.490475 (+ 417us) server_negotiation.cc:299] Negotiation successful
0620 20:18:58.490646 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260620 20:18:58.496233 30139 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60198:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 20:18:58.505195 30139 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 20:18:58.536826 30637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:58.520178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.3:41719 (local address 127.0.0.1:38560)
0620 20:18:58.520967 (+ 789us) negotiation.cc:107] Waiting for socket to connect
0620 20:18:58.521010 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 20:18:58.521209 (+ 199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:18:58.524725 (+ 3516us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:18:58.524734 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:18:58.524767 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:18:58.525227 (+ 460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:58.525241 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:58.528682 (+ 3441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:58.528688 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:58.532299 (+ 3611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:58.532316 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:58.534462 (+ 2146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:58.534469 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:58.534682 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:58.534745 (+ 63us) client_negotiation.cc:769] Sending connection context
0620 20:18:58.534867 (+ 122us) client_negotiation.cc:240] Negotiation successful
0620 20:18:58.536421 (+ 1554us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":491,"thread_start_us":3595,"threads_started":1}
I20260620 20:18:58.538846 30640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:58.521591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38560 (local address 127.16.217.3:41719)
0620 20:18:58.524026 (+ 2435us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:58.524035 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:58.524068 (+ 33us) server_negotiation.cc:407] Connection header received
0620 20:18:58.524181 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:58.524189 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:58.524450 (+ 261us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:58.525451 (+ 1001us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:18:58.525752 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:58.528513 (+ 2761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:58.532537 (+ 4024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:58.534185 (+ 1648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:58.537967 (+ 3782us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:58.538240 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0620 20:18:58.538320 (+ 80us) server_negotiation.cc:299] Negotiation successful
0620 20:18:58.538502 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2252,"thread_start_us":2906,"threads_started":1}
I20260620 20:18:58.544198 30550 tablet_service.cc:1467] Processing CreateTablet for tablet 7cf45766f4154e9f8eda26ae6beee120 (DEFAULT_TABLE table=test-table [id=f51c2c89755245268f677187f8a14db2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:18:58.546538 30550 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7cf45766f4154e9f8eda26ae6beee120. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:18:58.561444 30637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:58.520866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.2:38689 (local address 127.0.0.1:60234)
0620 20:18:58.537651 (+ 16785us) negotiation.cc:107] Waiting for socket to connect
0620 20:18:58.537671 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 20:18:58.537781 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:18:58.543545 (+ 5764us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:18:58.543551 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:18:58.543578 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:18:58.543764 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:58.543775 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:58.552578 (+ 8803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:58.552583 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:58.554423 (+ 1840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:58.554440 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:58.560593 (+ 6153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:58.560600 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:58.560815 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:58.560868 (+ 53us) client_negotiation.cc:769] Sending connection context
0620 20:18:58.560942 (+ 74us) client_negotiation.cc:240] Negotiation successful
0620 20:18:58.561090 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16658,"thread_start_us":37752,"threads_started":1}
I20260620 20:18:58.562382 30641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:58.522560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43964 (local address 127.16.217.1:38219)
0620 20:18:58.539709 (+ 17149us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:58.539719 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:58.542665 (+ 2946us) server_negotiation.cc:407] Connection header received
0620 20:18:58.542773 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:58.542781 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:58.543037 (+ 256us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:58.543255 (+ 218us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:18:58.545240 (+ 1985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:58.549834 (+ 4594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:58.556454 (+ 6620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:58.559988 (+ 3534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:58.560186 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:58.560445 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0620 20:18:58.561801 (+ 1356us) server_negotiation.cc:299] Negotiation successful
0620 20:18:58.562015 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16964,"thread_start_us":24422,"threads_started":1}
I20260620 20:18:58.561179 30639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:58.520996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.1:38219 (local address 127.0.0.1:43964)
0620 20:18:58.542378 (+ 21382us) negotiation.cc:107] Waiting for socket to connect
0620 20:18:58.542399 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 20:18:58.542516 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:18:58.543893 (+ 1377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:18:58.543897 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:18:58.543920 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:18:58.544292 (+ 372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:58.544303 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:58.550034 (+ 5731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:58.550040 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:58.552361 (+ 2321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:58.552395 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:58.560277 (+ 7882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:58.560284 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:58.560465 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:58.560504 (+ 39us) client_negotiation.cc:769] Sending connection context
0620 20:18:58.560618 (+ 114us) client_negotiation.cc:240] Negotiation successful
0620 20:18:58.560794 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21257,"thread_start_us":28988,"threads_started":1}
I20260620 20:18:58.562849 30642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:58.523127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60234 (local address 127.16.217.2:38689)
0620 20:18:58.540143 (+ 17016us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:58.540152 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:58.540193 (+ 41us) server_negotiation.cc:407] Connection header received
0620 20:18:58.540301 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:58.540310 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:58.540576 (+ 266us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:58.544516 (+ 3940us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:18:58.544772 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:58.547967 (+ 3195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:58.554649 (+ 6682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:58.556265 (+ 1616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:58.561322 (+ 5057us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:58.561936 (+ 614us) server_negotiation.cc:1035] Waiting for connection context
0620 20:18:58.562011 (+ 75us) server_negotiation.cc:299] Negotiation successful
0620 20:18:58.562203 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16837,"thread_start_us":23661,"threads_started":1}
I20260620 20:18:58.566820 30271 tablet_service.cc:1467] Processing CreateTablet for tablet 7cf45766f4154e9f8eda26ae6beee120 (DEFAULT_TABLE table=test-table [id=f51c2c89755245268f677187f8a14db2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:18:58.566828 30415 tablet_service.cc:1467] Processing CreateTablet for tablet 7cf45766f4154e9f8eda26ae6beee120 (DEFAULT_TABLE table=test-table [id=f51c2c89755245268f677187f8a14db2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:18:58.568624 30415 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7cf45766f4154e9f8eda26ae6beee120. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:18:58.569962 30271 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7cf45766f4154e9f8eda26ae6beee120. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:18:58.575469 30643 tablet_bootstrap.cc:492] T 7cf45766f4154e9f8eda26ae6beee120 P e3dc149d970842be80b46d8220cdf262: Bootstrap starting.
I20260620 20:18:58.579392 30643 tablet_bootstrap.cc:654] T 7cf45766f4154e9f8eda26ae6beee120 P e3dc149d970842be80b46d8220cdf262: Neither blocks nor log segments found. Creating new log.
I20260620 20:18:58.580807 30643 log.cc:824] T 7cf45766f4154e9f8eda26ae6beee120 P e3dc149d970842be80b46d8220cdf262: Log is configured to *not* fsync() on all Append() calls
I20260620 20:18:58.591364 30643 tablet_bootstrap.cc:492] T 7cf45766f4154e9f8eda26ae6beee120 P e3dc149d970842be80b46d8220cdf262: No bootstrap required, opened a new log
I20260620 20:18:58.591776 30643 ts_tablet_manager.cc:1397] T 7cf45766f4154e9f8eda26ae6beee120 P e3dc149d970842be80b46d8220cdf262: Time spent bootstrapping tablet: real 0.017s user 0.009s sys 0.004s
I20260620 20:18:58.598726 30645 tablet_bootstrap.cc:492] T 7cf45766f4154e9f8eda26ae6beee120 P 42865a9662cc44028f95ae95ee959729: Bootstrap starting.
I20260620 20:18:58.602679 30646 tablet_bootstrap.cc:492] T 7cf45766f4154e9f8eda26ae6beee120 P 890d1d3512c14e0288ac4e0ce29ce66d: Bootstrap starting.
I20260620 20:18:58.605031 30643 raft_consensus.cc:348] T 7cf45766f4154e9f8eda26ae6beee120 P e3dc149d970842be80b46d8220cdf262 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3dc149d970842be80b46d8220cdf262" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41719 } } peers { permanent_uuid: "890d1d3512c14e0288ac4e0ce29ce66d" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 38219 } } peers { permanent_uuid: "42865a9662cc44028f95ae95ee959729" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 38689 } }
I20260620 20:18:58.605430 30645 tablet_bootstrap.cc:654] T 7cf45766f4154e9f8eda26ae6beee120 P 42865a9662cc44028f95ae95ee959729: Neither blocks nor log segments found. Creating new log.
I20260620 20:18:58.605527 30643 raft_consensus.cc:374] T 7cf45766f4154e9f8eda26ae6beee120 P e3dc149d970842be80b46d8220cdf262 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:18:58.605620 30643 raft_consensus.cc:729] T 7cf45766f4154e9f8eda26ae6beee120 P e3dc149d970842be80b46d8220cdf262 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3dc149d970842be80b46d8220cdf262, State: Initialized, Role: FOLLOWER
I20260620 20:18:58.606324 30646 tablet_bootstrap.cc:654] T 7cf45766f4154e9f8eda26ae6beee120 P 890d1d3512c14e0288ac4e0ce29ce66d: Neither blocks nor log segments found. Creating new log.
I20260620 20:18:58.606398 30643 consensus_queue.cc:260] T 7cf45766f4154e9f8eda26ae6beee120 P e3dc149d970842be80b46d8220cdf262 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3dc149d970842be80b46d8220cdf262" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41719 } } peers { permanent_uuid: "890d1d3512c14e0288ac4e0ce29ce66d" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 38219 } } peers { permanent_uuid: "42865a9662cc44028f95ae95ee959729" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 38689 } }
I20260620 20:18:58.606805 30645 log.cc:824] T 7cf45766f4154e9f8eda26ae6beee120 P 42865a9662cc44028f95ae95ee959729: Log is configured to *not* fsync() on all Append() calls
I20260620 20:18:58.607518 30646 log.cc:824] T 7cf45766f4154e9f8eda26ae6beee120 P 890d1d3512c14e0288ac4e0ce29ce66d: Log is configured to *not* fsync() on all Append() calls
I20260620 20:18:58.615029 30643 ts_tablet_manager.cc:1428] T 7cf45766f4154e9f8eda26ae6beee120 P e3dc149d970842be80b46d8220cdf262: Time spent starting tablet: real 0.023s user 0.014s sys 0.007s
I20260620 20:18:58.615803 30615 heartbeater.cc:503] Master 127.16.217.62:40353 was elected leader, sending a full tablet report...
I20260620 20:18:58.618584 30646 tablet_bootstrap.cc:492] T 7cf45766f4154e9f8eda26ae6beee120 P 890d1d3512c14e0288ac4e0ce29ce66d: No bootstrap required, opened a new log
I20260620 20:18:58.618908 30645 tablet_bootstrap.cc:492] T 7cf45766f4154e9f8eda26ae6beee120 P 42865a9662cc44028f95ae95ee959729: No bootstrap required, opened a new log
I20260620 20:18:58.618925 30646 ts_tablet_manager.cc:1397] T 7cf45766f4154e9f8eda26ae6beee120 P 890d1d3512c14e0288ac4e0ce29ce66d: Time spent bootstrapping tablet: real 0.016s user 0.010s sys 0.005s
I20260620 20:18:58.619177 30645 ts_tablet_manager.cc:1397] T 7cf45766f4154e9f8eda26ae6beee120 P 42865a9662cc44028f95ae95ee959729: Time spent bootstrapping tablet: real 0.021s user 0.001s sys 0.014s
I20260620 20:18:58.627562 30645 raft_consensus.cc:348] T 7cf45766f4154e9f8eda26ae6beee120 P 42865a9662cc44028f95ae95ee959729 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3dc149d970842be80b46d8220cdf262" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41719 } } peers { permanent_uuid: "890d1d3512c14e0288ac4e0ce29ce66d" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 38219 } } peers { permanent_uuid: "42865a9662cc44028f95ae95ee959729" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 38689 } }
I20260620 20:18:58.627995 30645 raft_consensus.cc:374] T 7cf45766f4154e9f8eda26ae6beee120 P 42865a9662cc44028f95ae95ee959729 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:18:58.628139 30645 raft_consensus.cc:729] T 7cf45766f4154e9f8eda26ae6beee120 P 42865a9662cc44028f95ae95ee959729 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42865a9662cc44028f95ae95ee959729, State: Initialized, Role: FOLLOWER
I20260620 20:18:58.628839 30645 consensus_queue.cc:260] T 7cf45766f4154e9f8eda26ae6beee120 P 42865a9662cc44028f95ae95ee959729 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3dc149d970842be80b46d8220cdf262" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41719 } } peers { permanent_uuid: "890d1d3512c14e0288ac4e0ce29ce66d" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 38219 } } peers { permanent_uuid: "42865a9662cc44028f95ae95ee959729" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 38689 } }
I20260620 20:18:58.629349 30646 raft_consensus.cc:348] T 7cf45766f4154e9f8eda26ae6beee120 P 890d1d3512c14e0288ac4e0ce29ce66d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3dc149d970842be80b46d8220cdf262" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41719 } } peers { permanent_uuid: "890d1d3512c14e0288ac4e0ce29ce66d" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 38219 } } peers { permanent_uuid: "42865a9662cc44028f95ae95ee959729" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 38689 } }
I20260620 20:18:58.629797 30646 raft_consensus.cc:374] T 7cf45766f4154e9f8eda26ae6beee120 P 890d1d3512c14e0288ac4e0ce29ce66d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:18:58.629943 30646 raft_consensus.cc:729] T 7cf45766f4154e9f8eda26ae6beee120 P 890d1d3512c14e0288ac4e0ce29ce66d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 890d1d3512c14e0288ac4e0ce29ce66d, State: Initialized, Role: FOLLOWER
I20260620 20:18:58.630595 30646 consensus_queue.cc:260] T 7cf45766f4154e9f8eda26ae6beee120 P 890d1d3512c14e0288ac4e0ce29ce66d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3dc149d970842be80b46d8220cdf262" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41719 } } peers { permanent_uuid: "890d1d3512c14e0288ac4e0ce29ce66d" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 38219 } } peers { permanent_uuid: "42865a9662cc44028f95ae95ee959729" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 38689 } }
I20260620 20:18:58.630820 30645 ts_tablet_manager.cc:1428] T 7cf45766f4154e9f8eda26ae6beee120 P 42865a9662cc44028f95ae95ee959729: Time spent starting tablet: real 0.011s user 0.011s sys 0.001s
I20260620 20:18:58.631937 30336 heartbeater.cc:503] Master 127.16.217.62:40353 was elected leader, sending a full tablet report...
I20260620 20:18:58.632488 30646 ts_tablet_manager.cc:1428] T 7cf45766f4154e9f8eda26ae6beee120 P 890d1d3512c14e0288ac4e0ce29ce66d: Time spent starting tablet: real 0.013s user 0.010s sys 0.004s
W20260620 20:18:58.683660 30337 tablet.cc:2369] T 7cf45766f4154e9f8eda26ae6beee120 P 890d1d3512c14e0288ac4e0ce29ce66d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 20:18:58.688246 30616 tablet.cc:2369] T 7cf45766f4154e9f8eda26ae6beee120 P e3dc149d970842be80b46d8220cdf262: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 20:18:58.730374 30647 raft_consensus.cc:482] T 7cf45766f4154e9f8eda26ae6beee120 P e3dc149d970842be80b46d8220cdf262 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 20:18:58.730793 30647 raft_consensus.cc:504] T 7cf45766f4154e9f8eda26ae6beee120 P e3dc149d970842be80b46d8220cdf262 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3dc149d970842be80b46d8220cdf262" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41719 } } peers { permanent_uuid: "890d1d3512c14e0288ac4e0ce29ce66d" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 38219 } } peers { permanent_uuid: "42865a9662cc44028f95ae95ee959729" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 38689 } }
I20260620 20:18:58.732649 30647 leader_election.cc:288] T 7cf45766f4154e9f8eda26ae6beee120 P e3dc149d970842be80b46d8220cdf262 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 890d1d3512c14e0288ac4e0ce29ce66d (127.16.217.1:38219), 42865a9662cc44028f95ae95ee959729 (127.16.217.2:38689)
I20260620 20:18:58.739653 30617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:58.732890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.1:38219 (local address 127.16.217.3:36193)
0620 20:18:58.733312 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0620 20:18:58.733332 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 20:18:58.733462 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:18:58.734149 (+ 687us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:18:58.734157 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:18:58.734185 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:18:58.734395 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:58.734403 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:58.735458 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:58.735464 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:58.737272 (+ 1808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:58.737284 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:58.738846 (+ 1562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:58.738852 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:58.739067 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:58.739111 (+ 44us) client_negotiation.cc:769] Sending connection context
0620 20:18:58.739200 (+ 89us) client_negotiation.cc:240] Negotiation successful
0620 20:18:58.739342 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285,"mutex_wait_us":6}
I20260620 20:18:58.739645 30641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:58.733570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.217.3:36193 (local address 127.16.217.1:38219)
0620 20:18:58.733799 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:58.733804 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:58.733824 (+ 20us) server_negotiation.cc:407] Connection header received
0620 20:18:58.733920 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:58.733925 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:58.733998 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:58.734081 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:18:58.734543 (+ 462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:58.735302 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:58.737516 (+ 2214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:58.738668 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:58.738837 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:58.738910 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 20:18:58.739274 (+ 364us) server_negotiation.cc:299] Negotiation successful
0620 20:18:58.739404 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 20:18:58.740808 30291 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cf45766f4154e9f8eda26ae6beee120" candidate_uuid: "e3dc149d970842be80b46d8220cdf262" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "890d1d3512c14e0288ac4e0ce29ce66d" is_pre_election: true
I20260620 20:18:58.741357 30642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:58.733240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.217.3:49247 (local address 127.16.217.2:38689)
0620 20:18:58.733466 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:58.733473 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:58.734059 (+ 586us) server_negotiation.cc:407] Connection header received
0620 20:18:58.734326 (+ 267us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:58.734332 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:58.734400 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:58.734513 (+ 113us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:18:58.736133 (+ 1620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:58.737259 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:58.739252 (+ 1993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:58.740383 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:58.740535 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:58.740601 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 20:18:58.740966 (+ 365us) server_negotiation.cc:299] Negotiation successful
0620 20:18:58.741099 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260620 20:18:58.741451 30652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:58.733138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.2:38689 (local address 127.16.217.3:49247)
0620 20:18:58.733947 (+ 809us) negotiation.cc:107] Waiting for socket to connect
0620 20:18:58.733962 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 20:18:58.734125 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:18:58.735487 (+ 1362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:18:58.735494 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:18:58.735529 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:18:58.735943 (+ 414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:58.735954 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:58.737420 (+ 1466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:58.737426 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:58.739048 (+ 1622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:58.739061 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:58.740583 (+ 1522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:58.740590 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:58.740773 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:58.740816 (+ 43us) client_negotiation.cc:769] Sending connection context
0620 20:18:58.740922 (+ 106us) client_negotiation.cc:240] Negotiation successful
0620 20:18:58.741071 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":661,"mutex_wait_us":26,"thread_start_us":500,"threads_started":1}
I20260620 20:18:58.741446 30291 raft_consensus.cc:2454] T 7cf45766f4154e9f8eda26ae6beee120 P 890d1d3512c14e0288ac4e0ce29ce66d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e3dc149d970842be80b46d8220cdf262 in term 0.
I20260620 20:18:58.742269 30505 leader_election.cc:302] T 7cf45766f4154e9f8eda26ae6beee120 P e3dc149d970842be80b46d8220cdf262 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 890d1d3512c14e0288ac4e0ce29ce66d, e3dc149d970842be80b46d8220cdf262; no voters:
I20260620 20:18:58.742493 30435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cf45766f4154e9f8eda26ae6beee120" candidate_uuid: "e3dc149d970842be80b46d8220cdf262" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42865a9662cc44028f95ae95ee959729" is_pre_election: true
I20260620 20:18:58.742702 30647 raft_consensus.cc:2781] T 7cf45766f4154e9f8eda26ae6beee120 P e3dc149d970842be80b46d8220cdf262 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 20:18:58.742857 30647 raft_consensus.cc:482] T 7cf45766f4154e9f8eda26ae6beee120 P e3dc149d970842be80b46d8220cdf262 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 20:18:58.742986 30647 raft_consensus.cc:3037] T 7cf45766f4154e9f8eda26ae6beee120 P e3dc149d970842be80b46d8220cdf262 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:18:58.743032 30435 raft_consensus.cc:2454] T 7cf45766f4154e9f8eda26ae6beee120 P 42865a9662cc44028f95ae95ee959729 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e3dc149d970842be80b46d8220cdf262 in term 0.
I20260620 20:18:58.745807 30647 raft_consensus.cc:504] T 7cf45766f4154e9f8eda26ae6beee120 P e3dc149d970842be80b46d8220cdf262 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3dc149d970842be80b46d8220cdf262" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41719 } } peers { permanent_uuid: "890d1d3512c14e0288ac4e0ce29ce66d" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 38219 } } peers { permanent_uuid: "42865a9662cc44028f95ae95ee959729" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 38689 } }
I20260620 20:18:58.746572 30647 leader_election.cc:288] T 7cf45766f4154e9f8eda26ae6beee120 P e3dc149d970842be80b46d8220cdf262 [CANDIDATE]: Term 1 election: Requested vote from peers 890d1d3512c14e0288ac4e0ce29ce66d (127.16.217.1:38219), 42865a9662cc44028f95ae95ee959729 (127.16.217.2:38689)
I20260620 20:18:58.747025 30291 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cf45766f4154e9f8eda26ae6beee120" candidate_uuid: "e3dc149d970842be80b46d8220cdf262" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "890d1d3512c14e0288ac4e0ce29ce66d"
I20260620 20:18:58.747061 30435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cf45766f4154e9f8eda26ae6beee120" candidate_uuid: "e3dc149d970842be80b46d8220cdf262" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42865a9662cc44028f95ae95ee959729"
I20260620 20:18:58.747273 30291 raft_consensus.cc:3037] T 7cf45766f4154e9f8eda26ae6beee120 P 890d1d3512c14e0288ac4e0ce29ce66d [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:18:58.747277 30435 raft_consensus.cc:3037] T 7cf45766f4154e9f8eda26ae6beee120 P 42865a9662cc44028f95ae95ee959729 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:18:58.749562 30291 raft_consensus.cc:2454] T 7cf45766f4154e9f8eda26ae6beee120 P 890d1d3512c14e0288ac4e0ce29ce66d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e3dc149d970842be80b46d8220cdf262 in term 1.
I20260620 20:18:58.750046 30505 leader_election.cc:302] T 7cf45766f4154e9f8eda26ae6beee120 P e3dc149d970842be80b46d8220cdf262 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 890d1d3512c14e0288ac4e0ce29ce66d, e3dc149d970842be80b46d8220cdf262; no voters:
I20260620 20:18:58.750332 30647 raft_consensus.cc:2781] T 7cf45766f4154e9f8eda26ae6beee120 P e3dc149d970842be80b46d8220cdf262 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:18:58.750373 30435 raft_consensus.cc:2454] T 7cf45766f4154e9f8eda26ae6beee120 P 42865a9662cc44028f95ae95ee959729 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e3dc149d970842be80b46d8220cdf262 in term 1.
I20260620 20:18:58.757467 30647 raft_consensus.cc:686] T 7cf45766f4154e9f8eda26ae6beee120 P e3dc149d970842be80b46d8220cdf262 [term 1 LEADER]: Becoming Leader. State: Replica: e3dc149d970842be80b46d8220cdf262, State: Running, Role: LEADER
I20260620 20:18:58.758153 30647 consensus_queue.cc:237] T 7cf45766f4154e9f8eda26ae6beee120 P e3dc149d970842be80b46d8220cdf262 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3dc149d970842be80b46d8220cdf262" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41719 } } peers { permanent_uuid: "890d1d3512c14e0288ac4e0ce29ce66d" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 38219 } } peers { permanent_uuid: "42865a9662cc44028f95ae95ee959729" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 38689 } }
I20260620 20:18:58.763581 30139 catalog_manager.cc:5495] T 7cf45766f4154e9f8eda26ae6beee120 P e3dc149d970842be80b46d8220cdf262 reported cstate change: term changed from 0 to 1, leader changed from <none> to e3dc149d970842be80b46d8220cdf262 (127.16.217.3). New cstate: current_term: 1 leader_uuid: "e3dc149d970842be80b46d8220cdf262" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3dc149d970842be80b46d8220cdf262" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41719 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "890d1d3512c14e0288ac4e0ce29ce66d" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 38219 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "42865a9662cc44028f95ae95ee959729" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 38689 } health_report { overall_health: UNKNOWN } } }
I20260620 20:18:58.829605 30640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:58.823568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38576 (local address 127.16.217.3:41719)
0620 20:18:58.823794 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:58.823800 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:58.823821 (+ 21us) server_negotiation.cc:407] Connection header received
0620 20:18:58.823925 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:58.823931 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:58.824027 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:58.824183 (+ 156us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 20:18:58.824501 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:58.825592 (+ 1091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:58.827126 (+ 1534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:58.827745 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:58.827867 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:58.828388 (+ 521us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 20:18:58.829049 (+ 661us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 20:18:58.829200 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0620 20:18:58.829255 (+ 55us) server_negotiation.cc:299] Negotiation successful
0620 20:18:58.829336 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260620 20:18:58.838238 30435 raft_consensus.cc:1261] T 7cf45766f4154e9f8eda26ae6beee120 P 42865a9662cc44028f95ae95ee959729 [term 1 FOLLOWER]: Refusing update from remote peer e3dc149d970842be80b46d8220cdf262: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:18:58.838302 30291 raft_consensus.cc:1261] T 7cf45766f4154e9f8eda26ae6beee120 P 890d1d3512c14e0288ac4e0ce29ce66d [term 1 FOLLOWER]: Refusing update from remote peer e3dc149d970842be80b46d8220cdf262: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:18:58.839283 30653 consensus_queue.cc:1035] T 7cf45766f4154e9f8eda26ae6beee120 P e3dc149d970842be80b46d8220cdf262 [LEADER]: Connected to new peer: Peer: permanent_uuid: "890d1d3512c14e0288ac4e0ce29ce66d" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 38219 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:18:58.840010 30647 consensus_queue.cc:1035] T 7cf45766f4154e9f8eda26ae6beee120 P e3dc149d970842be80b46d8220cdf262 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42865a9662cc44028f95ae95ee959729" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 38689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:18:58.854907 30656 mvcc.cc:205] Tried to move back new op lower bound from 7299017682270363648 to 7299017681957625856. Current Snapshot: MvccSnapshot[applied={T|T < 7299017682270363648}]
I20260620 20:18:58.863368 30658 mvcc.cc:205] Tried to move back new op lower bound from 7299017682270363648 to 7299017681957625856. Current Snapshot: MvccSnapshot[applied={T|T < 7299017682270363648}]
I20260620 20:18:58.870409 30139 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60198:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 20:18:58.870687 30139 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60198:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 20:18:58.874855 30139 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 12eb0dcc6c7f4782be00f8df729c5aba: Sending DeleteTablet for 3 replicas of tablet 7cf45766f4154e9f8eda26ae6beee120
I20260620 20:18:58.876080 30271 tablet_service.cc:1514] Processing DeleteTablet for tablet 7cf45766f4154e9f8eda26ae6beee120 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:18:58 UTC) from {username='kudu', principal='kudu/127.16.217.62@KRBTEST.COM'} at 127.0.0.1:43964
I20260620 20:18:58.876080 30550 tablet_service.cc:1514] Processing DeleteTablet for tablet 7cf45766f4154e9f8eda26ae6beee120 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:18:58 UTC) from {username='kudu', principal='kudu/127.16.217.62@KRBTEST.COM'} at 127.0.0.1:38560
I20260620 20:18:58.876137 30415 tablet_service.cc:1514] Processing DeleteTablet for tablet 7cf45766f4154e9f8eda26ae6beee120 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:18:58 UTC) from {username='kudu', principal='kudu/127.16.217.62@KRBTEST.COM'} at 127.0.0.1:60234
I20260620 20:18:58.877096 30665 tablet_replica.cc:330] T 7cf45766f4154e9f8eda26ae6beee120 P e3dc149d970842be80b46d8220cdf262: stopping tablet replica
I20260620 20:18:58.877776 30665 raft_consensus.cc:2229] T 7cf45766f4154e9f8eda26ae6beee120 P e3dc149d970842be80b46d8220cdf262 [term 1 LEADER]: Raft consensus shutting down.
I20260620 20:18:58.878791 30665 raft_consensus.cc:2258] T 7cf45766f4154e9f8eda26ae6beee120 P e3dc149d970842be80b46d8220cdf262 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:18:58.880026 30666 tablet_replica.cc:330] T 7cf45766f4154e9f8eda26ae6beee120 P 890d1d3512c14e0288ac4e0ce29ce66d: stopping tablet replica
I20260620 20:18:58.880786 30666 raft_consensus.cc:2229] T 7cf45766f4154e9f8eda26ae6beee120 P 890d1d3512c14e0288ac4e0ce29ce66d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 20:18:58.881541 30666 raft_consensus.cc:2258] T 7cf45766f4154e9f8eda26ae6beee120 P 890d1d3512c14e0288ac4e0ce29ce66d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:18:58.883883 30666 ts_tablet_manager.cc:1905] T 7cf45766f4154e9f8eda26ae6beee120 P 890d1d3512c14e0288ac4e0ce29ce66d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:18:58.883888 30665 ts_tablet_manager.cc:1905] T 7cf45766f4154e9f8eda26ae6beee120 P e3dc149d970842be80b46d8220cdf262: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:18:58.884867 17252 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu pid 30176
I20260620 20:18:58.891474 30666 ts_tablet_manager.cc:1918] T 7cf45766f4154e9f8eda26ae6beee120 P 890d1d3512c14e0288ac4e0ce29ce66d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 20:18:58.891634 30666 log.cc:1196] T 7cf45766f4154e9f8eda26ae6beee120 P 890d1d3512c14e0288ac4e0ce29ce66d: Deleting WAL directory at /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986536687152-17252-0/minicluster-data/ts-0/wal/wals/7cf45766f4154e9f8eda26ae6beee120
I20260620 20:18:58.892220 30666 ts_tablet_manager.cc:1939] T 7cf45766f4154e9f8eda26ae6beee120 P 890d1d3512c14e0288ac4e0ce29ce66d: Deleting consensus metadata
I20260620 20:18:58.892299 30667 tablet_replica.cc:330] T 7cf45766f4154e9f8eda26ae6beee120 P 42865a9662cc44028f95ae95ee959729: stopping tablet replica
I20260620 20:18:58.892755 30667 raft_consensus.cc:2229] T 7cf45766f4154e9f8eda26ae6beee120 P 42865a9662cc44028f95ae95ee959729 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 20:18:58.894213 30126 catalog_manager.cc:4841] TS 890d1d3512c14e0288ac4e0ce29ce66d (127.16.217.1:38219): tablet 7cf45766f4154e9f8eda26ae6beee120 (table test-table [id=f51c2c89755245268f677187f8a14db2]) successfully deleted
I20260620 20:18:58.896121 30667 raft_consensus.cc:2258] T 7cf45766f4154e9f8eda26ae6beee120 P 42865a9662cc44028f95ae95ee959729 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:18:58.898134 30665 ts_tablet_manager.cc:1918] T 7cf45766f4154e9f8eda26ae6beee120 P e3dc149d970842be80b46d8220cdf262: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 20:18:58.898396 30665 log.cc:1196] T 7cf45766f4154e9f8eda26ae6beee120 P e3dc149d970842be80b46d8220cdf262: Deleting WAL directory at /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986536687152-17252-0/minicluster-data/ts-2/wal/wals/7cf45766f4154e9f8eda26ae6beee120
I20260620 20:18:58.899063 30665 ts_tablet_manager.cc:1939] T 7cf45766f4154e9f8eda26ae6beee120 P e3dc149d970842be80b46d8220cdf262: Deleting consensus metadata
I20260620 20:18:58.899168 30667 ts_tablet_manager.cc:1905] T 7cf45766f4154e9f8eda26ae6beee120 P 42865a9662cc44028f95ae95ee959729: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:18:58.901079 30123 catalog_manager.cc:4841] TS e3dc149d970842be80b46d8220cdf262 (127.16.217.3:41719): tablet 7cf45766f4154e9f8eda26ae6beee120 (table test-table [id=f51c2c89755245268f677187f8a14db2]) successfully deleted
I20260620 20:18:58.903851 30641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:58.886337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43978 (local address 127.16.217.1:38219)
0620 20:18:58.886588 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:58.886599 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:58.886627 (+ 28us) server_negotiation.cc:407] Connection header received
0620 20:18:58.886726 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:58.886741 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:58.886850 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:58.887008 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:18:58.888722 (+ 1714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:58.890294 (+ 1572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:58.892190 (+ 1896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:58.892898 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:58.893061 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:58.895833 (+ 2772us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:18:58.895846 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:18:58.895864 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:18:58.895898 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:18:58.899957 (+ 4059us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:58.900646 (+ 689us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:58.900654 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:58.900658 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:58.900811 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:58.901156 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:58.901164 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:58.901166 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:58.902418 (+ 1252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:18:58.902800 (+ 382us) server_negotiation.cc:1035] Waiting for connection context
0620 20:18:58.903230 (+ 430us) server_negotiation.cc:299] Negotiation successful
0620 20:18:58.903495 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260620 20:18:58.906411 30331 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:18:58.909935 30667 ts_tablet_manager.cc:1918] T 7cf45766f4154e9f8eda26ae6beee120 P 42865a9662cc44028f95ae95ee959729: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 20:18:58.910156 30667 log.cc:1196] T 7cf45766f4154e9f8eda26ae6beee120 P 42865a9662cc44028f95ae95ee959729: Deleting WAL directory at /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986536687152-17252-0/minicluster-data/ts-1/wal/wals/7cf45766f4154e9f8eda26ae6beee120
I20260620 20:18:58.910745 30667 ts_tablet_manager.cc:1939] T 7cf45766f4154e9f8eda26ae6beee120 P 42865a9662cc44028f95ae95ee959729: Deleting consensus metadata
I20260620 20:18:58.912914 30124 catalog_manager.cc:4841] TS 42865a9662cc44028f95ae95ee959729 (127.16.217.2:38689): tablet 7cf45766f4154e9f8eda26ae6beee120 (table test-table [id=f51c2c89755245268f677187f8a14db2]) successfully deleted
I20260620 20:18:59.003927 17252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu with pid 30176
I20260620 20:18:59.021755 17252 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu pid 30340
I20260620 20:18:59.038196 30642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:59.022734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60244 (local address 127.16.217.2:38689)
0620 20:18:59.022922 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:59.022926 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:59.022951 (+ 25us) server_negotiation.cc:407] Connection header received
0620 20:18:59.023187 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:59.023191 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:59.023261 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:59.023356 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:18:59.025082 (+ 1726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:59.025843 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:59.027761 (+ 1918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:59.028148 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:59.028241 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:59.031479 (+ 3238us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:18:59.031494 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:18:59.031510 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:18:59.031543 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:18:59.035111 (+ 3568us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:59.035944 (+ 833us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:59.035952 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:59.035955 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:59.036108 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:59.036517 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:59.036521 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:59.036523 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:59.036885 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:18:59.037166 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0620 20:18:59.037520 (+ 354us) server_negotiation.cc:299] Negotiation successful
0620 20:18:59.037751 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 20:18:59.038887 30475 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:18:59.132093 17252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu with pid 30340
I20260620 20:18:59.149113 17252 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu pid 30485
I20260620 20:18:59.165459 30640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:59.149979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38588 (local address 127.16.217.3:41719)
0620 20:18:59.150224 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:59.150230 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:59.150254 (+ 24us) server_negotiation.cc:407] Connection header received
0620 20:18:59.150327 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:59.150333 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:59.150424 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:59.150546 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:18:59.151798 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:59.153011 (+ 1213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:59.154713 (+ 1702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:59.155438 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:59.155600 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:59.158304 (+ 2704us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:18:59.158320 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:18:59.158341 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:18:59.158396 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:18:59.162473 (+ 4077us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:59.163233 (+ 760us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:59.163242 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:59.163245 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:59.163418 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:59.163795 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:59.163802 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:59.163806 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:59.164270 (+ 464us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:18:59.164587 (+ 317us) server_negotiation.cc:1035] Waiting for connection context
0620 20:18:59.164796 (+ 209us) server_negotiation.cc:299] Negotiation successful
0620 20:18:59.165036 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108}
I20260620 20:18:59.166311 30610 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:18:59.289414 17252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu with pid 30485
I20260620 20:18:59.308444 17252 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu pid 30106
I20260620 20:18:59.309674 30169 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:18:59.422004 17252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu with pid 30106
I20260620 20:18:59.449005 17252 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260620 20:18:59.795048 17252 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260620 20:18:59.795297 17252 mini_ranger.cc:66] Stopping Ranger...
I20260620 20:19:00.141042 17252 mini_ranger.cc:68] Stopped Ranger
2026-06-20 20:19:00.141 UTC [29158] LOG: received smart shutdown request
2026-06-20 20:19:00.144 UTC [29158] LOG: background worker "logical replication launcher" (PID 29167) exited with exit code 1
2026-06-20 20:19:00.144 UTC [29162] LOG: shutting down
2026-06-20 20:19:00.327 UTC [29158] LOG: database system is shut down
2026-06-20T20:19:00Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (53477 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986536687152-17252-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 20:19:00 dist-test-slave-2hpg krb5kdc[30680](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 20:19:00 dist-test-slave-2hpg krb5kdc[30680](info): set up 2 sockets
Jun 20 20:19:00 dist-test-slave-2hpg krb5kdc[30680](info): commencing operation
krb5kdc: starting...
W20260620 20:19:01.570673 17252 mini_kdc.cc:121] Time spent starting KDC: real 1.068s user 0.001s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986536687152-17252-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986536687152-17252-0/krb5kdc/test-user.keytab.
Jun 20 20:19:01 dist-test-slave-2hpg krb5kdc[30680](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986741, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T20:19:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T20:19:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.217.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.217.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.217.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.62.keytab.
Entry for principal kudu/127.16.217.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.62.keytab.
I20260620 20:19:01.814688 17252 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
/tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986536687152-17252-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986536687152-17252-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986536687152-17252-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986536687152-17252-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.217.62:45423
--webserver_interface=127.16.217.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.62.keytab
--principal=kudu/127.16.217.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.217.20:34695
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.217.62:45423,127.16.217.61:41137,127.16.217.60:35767
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986536687152-17252-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986536687152-17252-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986536687152-17252-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:19:02.152333 30693 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:19:02.152705 30693 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:19:02.152861 30693 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:19:02.160290 30693 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:19:02.160451 30693 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:19:02.160570 30693 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:19:02.160676 30693 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:19:02.160758 30693 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:19:02.170810 30693 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.217.20:34695
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986536687152-17252-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986536687152-17252-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.217.62:45423,127.16.217.61:41137,127.16.217.60:35767
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.62.keytab
--principal=kudu/127.16.217.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.217.62:45423
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986536687152-17252-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.217.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986536687152-17252-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:19:02.172256 30693 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:19:02.173658 30693 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:19:02.185736 30698 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:19:02.185776 30699 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:19:02.185853 30701 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:19:02.186510 30693 server_base.cc:1034] running on GCE node
I20260620 20:19:02.187270 30693 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:19:02.188732 30693 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:19:02.189980 30693 hybrid_clock.cc:648] HybridClock initialized: now 1781986742189935 us; error 47 us; skew 500 ppm
I20260620 20:19:02.190371 30693 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:19:02 dist-test-slave-2hpg krb5kdc[30680](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986742, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.217.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:19:02.198365 30693 init.cc:377] Logged in from keytab as kudu/127.16.217.62@KRBTEST.COM (short username kudu)
I20260620 20:19:02.200037 30693 webserver.cc:460] Webserver started at http://127.16.217.62:40527/ using document root <none> and password file <none>
I20260620 20:19:02.200748 30693 fs_manager.cc:359] Metadata directory not provided
I20260620 20:19:02.200881 30693 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:19:02.201205 30693 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:19:02.204373 30693 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986536687152-17252-0/minicluster-data/master-0/data/instance:
uuid: "e27d0330bb6a43258dcba9fe7cf90eb1"
format_stamp: "Formatted at 2026-06-20 20:19:02 on dist-test-slave-2hpg"
server_key: "743264a4e8dad1d07fa348f4b7dab4ee"
server_key_iv: "879afbaf76cea4dd8c3b57d4544559ca"
server_key_version: "encryptionkey@0"
I20260620 20:19:02.205205 30693 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986536687152-17252-0/minicluster-data/master-0/wal/instance:
uuid: "e27d0330bb6a43258dcba9fe7cf90eb1"
format_stamp: "Formatted at 2026-06-20 20:19:02 on dist-test-slave-2hpg"
server_key: "743264a4e8dad1d07fa348f4b7dab4ee"
server_key_iv: "879afbaf76cea4dd8c3b57d4544559ca"
server_key_version: "encryptionkey@0"
I20260620 20:19:02.210517 30693 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260620 20:19:02.214308 30708 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:19:02.215991 30693 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 20:19:02.216216 30693 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986536687152-17252-0/minicluster-data/master-0/data,/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986536687152-17252-0/minicluster-data/master-0/wal
uuid: "e27d0330bb6a43258dcba9fe7cf90eb1"
format_stamp: "Formatted at 2026-06-20 20:19:02 on dist-test-slave-2hpg"
server_key: "743264a4e8dad1d07fa348f4b7dab4ee"
server_key_iv: "879afbaf76cea4dd8c3b57d4544559ca"
server_key_version: "encryptionkey@0"
I20260620 20:19:02.216451 30693 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986536687152-17252-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986536687152-17252-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986536687152-17252-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:19:02.240123 30693 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:19:02.245136 30693 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:19:02.245464 30693 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:19:02.266474 30693 rpc_server.cc:307] RPC server started. Bound to: 127.16.217.62:45423
I20260620 20:19:02.266522 30759 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.217.62:45423 every 8 connection(s)
I20260620 20:19:02.268035 30693 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986536687152-17252-0/minicluster-data/master-0/data/info.pb
I20260620 20:19:02.272117 30760 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:19:02.278115 17252 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu as pid 30693
I20260620 20:19:02.278365 17252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986536687152-17252-0/minicluster-data/master-0/wal/instance
I20260620 20:19:02.279155 17252 external_mini_cluster.cc:1428] Setting key 5e184e8ec2f0fbfa558962de9df09ec4
I20260620 20:19:02.279816 30760 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.217.62" port: 45423 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 20:19:02 dist-test-slave-2hpg krb5kdc[30680](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986742, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.217.62@KRBTEST.COM for kudu/127.16.217.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.16.217.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.217.61@KRBTEST.COM" created.
I20260620 20:19:02.307693 30762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:19:02.284238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:45423 (local address 127.0.0.1:33422)
0620 20:19:02.285146 (+ 908us) negotiation.cc:107] Waiting for socket to connect
0620 20:19:02.285214 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0620 20:19:02.288098 (+ 2884us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:19:02.289070 (+ 972us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:19:02.289085 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:19:02.289938 (+ 853us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:19:02.290497 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:19:02.290525 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:19:02.292234 (+ 1709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:19:02.292242 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:19:02.294247 (+ 2005us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:19:02.294265 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:19:02.295281 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:19:02.295291 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:19:02.295538 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:19:02.296271 (+ 733us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:19:02.296307 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:19:02.299831 (+ 3524us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:19:02.303425 (+ 3594us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:19:02.303435 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:19:02.303445 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:19:02.303877 (+ 432us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:19:02.304377 (+ 500us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:19:02.304383 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:19:02.304387 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:19:02.304633 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:19:02.305694 (+ 1061us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:19:02.305703 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:19:02.305969 (+ 266us) client_negotiation.cc:769] Sending connection context
0620 20:19:02.306205 (+ 236us) client_negotiation.cc:240] Negotiation successful
0620 20:19:02.306612 (+ 407us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":617,"thread_start_us":402,"threads_started":1}
I20260620 20:19:02.307739 30763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:19:02.285647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33422 (local address 127.16.217.62:45423)
0620 20:19:02.286449 (+ 802us) server_negotiation.cc:206] Beginning negotiation
0620 20:19:02.286469 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0620 20:19:02.286527 (+ 58us) server_negotiation.cc:407] Connection header received
0620 20:19:02.288549 (+ 2022us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:19:02.288556 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:19:02.288813 (+ 257us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:19:02.288993 (+ 180us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:19:02.290728 (+ 1735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:19:02.292044 (+ 1316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:19:02.294420 (+ 2376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:19:02.295014 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:19:02.295175 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:19:02.300148 (+ 4973us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:19:02.300157 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:19:02.300171 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:19:02.300203 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:19:02.303277 (+ 3074us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:19:02.304089 (+ 812us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:19:02.304102 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:19:02.304104 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:19:02.304262 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:19:02.304780 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:19:02.304786 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:19:02.304793 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:19:02.305541 (+ 748us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:19:02.305768 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0620 20:19:02.306400 (+ 632us) server_negotiation.cc:299] Negotiation successful
0620 20:19:02.306828 (+ 428us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":85,"server-negotiator.queue_time_us":557,"thread_start_us":369,"threads_started":1}
I20260620 20:19:02.310941 30760 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.217.61" port: 41137 } has no permanent_uuid. Determining permanent_uuid...
I20260620 20:19:02.312047 30762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:19:02.311594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.61:41137 (local address 127.0.0.1:59396)
0620 20:19:02.311807 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0620 20:19:02.311936 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.217.61:41137: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260620 20:19:02.312337 30709 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.61:41137: connect: Connection refused (error 111)
I20260620 20:19:02.314117 30762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:19:02.313577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.61:41137 (local address 127.0.0.1:59412)
0620 20:19:02.313792 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0620 20:19:02.313976 (+ 184us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.217.61:41137: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102}
W20260620 20:19:02.314308 30760 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.217.61:41137: Network error: Client connection negotiation failed: client connection to 127.16.217.61:41137: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.217.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.61.keytab.
Entry for principal kudu/127.16.217.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.61.keytab.
I20260620 20:19:02.333753 17252 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
/tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986536687152-17252-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986536687152-17252-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986536687152-17252-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986536687152-17252-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.217.61:41137
--webserver_interface=127.16.217.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.61.keytab
--principal=kudu/127.16.217.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.217.20:34695
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.217.62:45423,127.16.217.61:41137,127.16.217.60:35767
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986536687152-17252-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986536687152-17252-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986536687152-17252-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 20:19:02.363628 30760 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.217.61" port: 41137 } attempt: 1
I20260620 20:19:02.364866 30762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:19:02.364383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.61:41137 (local address 127.0.0.1:59420)
0620 20:19:02.364584 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0620 20:19:02.364728 (+ 144us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.217.61:41137: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260620 20:19:02.366022 30762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:19:02.365553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.61:41137 (local address 127.0.0.1:59430)
0620 20:19:02.365734 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0620 20:19:02.365825 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.217.61:41137: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260620 20:19:02.366331 30760 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.217.61:41137: Network error: Client connection negotiation failed: client connection to 127.16.217.61:41137: connect: Connection refused (error 111)
I20260620 20:19:02.434633 30760 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.217.61" port: 41137 } attempt: 2
I20260620 20:19:02.436014 30762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:19:02.435380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.61:41137 (local address 127.0.0.1:59436)
0620 20:19:02.435677 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0620 20:19:02.435807 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.217.61:41137: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260620 20:19:02.437106 30762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:19:02.436697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.61:41137 (local address 127.0.0.1:59446)
0620 20:19:02.436847 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0620 20:19:02.436947 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.217.61:41137: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260620 20:19:02.437305 30760 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.217.61:41137: Network error: Client connection negotiation failed: client connection to 127.16.217.61:41137: connect: Connection refused (error 111)
I20260620 20:19:02.528605 30760 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.217.61" port: 41137 } attempt: 3
I20260620 20:19:02.529843 30762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:19:02.529336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.61:41137 (local address 127.0.0.1:59450)
0620 20:19:02.529556 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0620 20:19:02.529679 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.217.61:41137: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260620 20:19:02.530982 30762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:19:02.530561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.61:41137 (local address 127.0.0.1:59456)
0620 20:19:02.530716 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0620 20:19:02.530820 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.217.61:41137: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260620 20:19:02.531239 30760 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.217.61:41137: Network error: Client connection negotiation failed: client connection to 127.16.217.61:41137: connect: Connection refused (error 111)
W20260620 20:19:02.663751 30767 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:19:02.664052 30767 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:19:02.664237 30767 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:19:02.671711 30767 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:19:02.671842 30767 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:19:02.671891 30767 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:19:02.672014 30767 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:19:02.672097 30767 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:19:02.674535 30760 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.217.61" port: 41137 } attempt: 4
I20260620 20:19:02.675855 30762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:19:02.675230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.61:41137 (local address 127.0.0.1:59460)
0620 20:19:02.675573 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0620 20:19:02.675685 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.217.61:41137: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175}
I20260620 20:19:02.677065 30762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:19:02.676621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.61:41137 (local address 127.0.0.1:59468)
0620 20:19:02.676788 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0620 20:19:02.676899 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.217.61:41137: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260620 20:19:02.677299 30760 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.217.61:41137: Network error: Client connection negotiation failed: client connection to 127.16.217.61:41137: connect: Connection refused (error 111)
I20260620 20:19:02.683292 30767 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.217.20:34695
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986536687152-17252-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986536687152-17252-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.16.217.62:45423,127.16.217.61:41137,127.16.217.60:35767
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.61.keytab
--principal=kudu/127.16.217.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.217.61:41137
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986536687152-17252-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.16.217.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986536687152-17252-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:19:02.684720 30767 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:19:02.686105 30767 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:19:02.697976 30775 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:19:02.697993 30773 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:19:02.697969 30772 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:19:02.699112 30767 server_base.cc:1034] running on GCE node
I20260620 20:19:02.699893 30767 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:19:02.701387 30767 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:19:02.702661 30767 hybrid_clock.cc:648] HybridClock initialized: now 1781986742702605 us; error 51 us; skew 500 ppm
I20260620 20:19:02.703023 30767 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:19:02 dist-test-slave-2hpg krb5kdc[30680](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986742, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.217.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:19:02.706111 30767 init.cc:377] Logged in from keytab as kudu/127.16.217.61@KRBTEST.COM (short username kudu)
I20260620 20:19:02.707648 30767 webserver.cc:460] Webserver started at http://127.16.217.61:39085/ using document root <none> and password file <none>
I20260620 20:19:02.708341 30767 fs_manager.cc:359] Metadata directory not provided
I20260620 20:19:02.708485 30767 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:19:02.708803 30767 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:19:02.711910 30767 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986536687152-17252-0/minicluster-data/master-1/data/instance:
uuid: "861816685d044c968d3fd1013f1fff55"
format_stamp: "Formatted at 2026-06-20 20:19:02 on dist-test-slave-2hpg"
server_key: "51c784927353abfa4ac83094b76e1791"
server_key_iv: "7466406134c2d9e9a7f8dcdd3c44fcc0"
server_key_version: "encryptionkey@0"
I20260620 20:19:02.712756 30767 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986536687152-17252-0/minicluster-data/master-1/wal/instance:
uuid: "861816685d044c968d3fd1013f1fff55"
format_stamp: "Formatted at 2026-06-20 20:19:02 on dist-test-slave-2hpg"
server_key: "51c784927353abfa4ac83094b76e1791"
server_key_iv: "7466406134c2d9e9a7f8dcdd3c44fcc0"
server_key_version: "encryptionkey@0"
I20260620 20:19:02.718174 30767 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260620 20:19:02.721990 30782 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:19:02.723474 30767 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 20:19:02.723706 30767 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986536687152-17252-0/minicluster-data/master-1/data,/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986536687152-17252-0/minicluster-data/master-1/wal
uuid: "861816685d044c968d3fd1013f1fff55"
format_stamp: "Formatted at 2026-06-20 20:19:02 on dist-test-slave-2hpg"
server_key: "51c784927353abfa4ac83094b76e1791"
server_key_iv: "7466406134c2d9e9a7f8dcdd3c44fcc0"
server_key_version: "encryptionkey@0"
I20260620 20:19:02.723923 30767 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986536687152-17252-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986536687152-17252-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986536687152-17252-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:19:02.746624 30767 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:19:02.750792 30767 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:19:02.751078 30767 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:19:02.770370 30767 rpc_server.cc:307] RPC server started. Bound to: 127.16.217.61:41137
I20260620 20:19:02.770414 30833 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.217.61:41137 every 8 connection(s)
I20260620 20:19:02.771814 30767 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986536687152-17252-0/minicluster-data/master-1/data/info.pb
I20260620 20:19:02.775612 30834 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:19:02.776150 17252 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu as pid 30767
I20260620 20:19:02.776335 17252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986536687152-17252-0/minicluster-data/master-1/wal/instance
I20260620 20:19:02.776789 17252 external_mini_cluster.cc:1428] Setting key 7bedaeb8597981d060e21abe9d443dbb
I20260620 20:19:02.782673 30834 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.217.62" port: 45423 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.16.217.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.217.60@KRBTEST.COM" created.
Jun 20 20:19:02 dist-test-slave-2hpg krb5kdc[30680](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986742, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.217.61@KRBTEST.COM for kudu/127.16.217.62@KRBTEST.COM
I20260620 20:19:02.807685 30763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:19:02.785698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33428 (local address 127.16.217.62:45423)
0620 20:19:02.785980 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0620 20:19:02.785989 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 20:19:02.787262 (+ 1273us) server_negotiation.cc:407] Connection header received
0620 20:19:02.789168 (+ 1906us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:19:02.789177 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:19:02.789278 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:19:02.789399 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:19:02.790985 (+ 1586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:19:02.792137 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:19:02.793822 (+ 1685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:19:02.794628 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:19:02.794775 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:19:02.800348 (+ 5573us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:19:02.800353 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:19:02.800356 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:19:02.800396 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:19:02.804009 (+ 3613us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:19:02.805162 (+ 1153us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:19:02.805170 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:19:02.805176 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:19:02.805316 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:19:02.805831 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:19:02.805836 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:19:02.805840 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:19:02.806202 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:19:02.806370 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0620 20:19:02.807128 (+ 758us) server_negotiation.cc:299] Negotiation successful
0620 20:19:02.807359 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260620 20:19:02.808702 30837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:19:02.786219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:45423 (local address 127.0.0.1:33428)
0620 20:19:02.787057 (+ 838us) negotiation.cc:107] Waiting for socket to connect
0620 20:19:02.787120 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0620 20:19:02.788883 (+ 1763us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:19:02.789593 (+ 710us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:19:02.789601 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:19:02.790223 (+ 622us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:19:02.790788 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:19:02.790805 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:19:02.792354 (+ 1549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:19:02.792359 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:19:02.793660 (+ 1301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:19:02.793672 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:19:02.794861 (+ 1189us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:19:02.794866 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:19:02.795053 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:19:02.795829 (+ 776us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:19:02.795857 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:19:02.800063 (+ 4206us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:19:02.804209 (+ 4146us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:19:02.804223 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:19:02.804247 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:19:02.805013 (+ 766us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:19:02.805460 (+ 447us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:19:02.805467 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:19:02.805472 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:19:02.805693 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:19:02.806349 (+ 656us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:19:02.806358 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:19:02.806735 (+ 377us) client_negotiation.cc:769] Sending connection context
0620 20:19:02.807019 (+ 284us) client_negotiation.cc:240] Negotiation successful
0620 20:19:02.807543 (+ 524us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":571,"thread_start_us":381,"threads_started":1}
I20260620 20:19:02.810469 30834 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.217.61" port: 41137 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 20:19:02 dist-test-slave-2hpg krb5kdc[30680](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986742, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.217.61@KRBTEST.COM for kudu/127.16.217.61@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.217.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.60.keytab.
Entry for principal kudu/127.16.217.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.60.keytab.
I20260620 20:19:02.826894 17252 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
/tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986536687152-17252-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986536687152-17252-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986536687152-17252-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986536687152-17252-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.217.60:35767
--webserver_interface=127.16.217.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.60.keytab
--principal=kudu/127.16.217.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.217.20:34695
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.217.62:45423,127.16.217.61:41137,127.16.217.60:35767
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986536687152-17252-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986536687152-17252-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986536687152-17252-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 20:19:02.827433 30839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:19:02.811423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59480 (local address 127.16.217.61:41137)
0620 20:19:02.812111 (+ 688us) server_negotiation.cc:206] Beginning negotiation
0620 20:19:02.812118 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:19:02.812154 (+ 36us) server_negotiation.cc:407] Connection header received
0620 20:19:02.812200 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:19:02.812205 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:19:02.812388 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:19:02.812578 (+ 190us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:19:02.813314 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:19:02.814483 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:19:02.815823 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:19:02.816388 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:19:02.816527 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:19:02.821587 (+ 5060us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:19:02.821600 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:19:02.821616 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:19:02.821650 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:19:02.824444 (+ 2794us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:19:02.825123 (+ 679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:19:02.825136 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:19:02.825139 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:19:02.825253 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:19:02.825627 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:19:02.825631 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:19:02.825635 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:19:02.826078 (+ 443us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:19:02.826305 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0620 20:19:02.826635 (+ 330us) server_negotiation.cc:299] Negotiation successful
0620 20:19:02.826930 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":60,"server-negotiator.queue_time_us":533,"thread_start_us":391,"threads_started":1}
I20260620 20:19:02.827441 30837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:19:02.811120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.61:41137 (local address 127.0.0.1:59480)
0620 20:19:02.811349 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0620 20:19:02.811364 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 20:19:02.811445 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:19:02.812555 (+ 1110us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:19:02.812560 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:19:02.813009 (+ 449us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:19:02.813157 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:19:02.813166 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:19:02.814668 (+ 1502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:19:02.814675 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:19:02.815615 (+ 940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:19:02.815627 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:19:02.816535 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:19:02.816540 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:19:02.816773 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:19:02.817409 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:19:02.817425 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:19:02.821365 (+ 3940us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:19:02.824599 (+ 3234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:19:02.824604 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:19:02.824608 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:19:02.824982 (+ 374us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:19:02.825362 (+ 380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:19:02.825367 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:19:02.825371 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:19:02.825513 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:19:02.826193 (+ 680us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:19:02.826200 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:19:02.826341 (+ 141us) client_negotiation.cc:769] Sending connection context
0620 20:19:02.826488 (+ 147us) client_negotiation.cc:240] Negotiation successful
0620 20:19:02.826790 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134}
I20260620 20:19:02.830401 30834 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.217.60" port: 35767 } has no permanent_uuid. Determining permanent_uuid...
I20260620 20:19:02.831691 30837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:19:02.831082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.60:35767 (local address 127.0.0.1:54418)
0620 20:19:02.831310 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0620 20:19:02.831464 (+ 154us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.217.60:35767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20260620 20:19:02.831950 30783 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.60:35767: connect: Connection refused (error 111)
I20260620 20:19:02.833648 30837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:19:02.833184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.60:35767 (local address 127.0.0.1:54424)
0620 20:19:02.833374 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0620 20:19:02.833481 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.217.60:35767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20260620 20:19:02.833884 30834 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.217.60:35767: Network error: Client connection negotiation failed: client connection to 127.16.217.60:35767: connect: Connection refused (error 111)
I20260620 20:19:02.883231 30834 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.217.60" port: 35767 } attempt: 1
I20260620 20:19:02.884539 30837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:19:02.884061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.60:35767 (local address 127.0.0.1:54438)
0620 20:19:02.884281 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0620 20:19:02.884389 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.217.60:35767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260620 20:19:02.885600 30837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:19:02.885269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.60:35767 (local address 127.0.0.1:54448)
0620 20:19:02.885432 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0620 20:19:02.885497 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.217.60:35767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260620 20:19:02.885805 30834 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.217.60:35767: Network error: Client connection negotiation failed: client connection to 127.16.217.60:35767: connect: Connection refused (error 111)
I20260620 20:19:02.954080 30834 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.217.60" port: 35767 } attempt: 2
I20260620 20:19:02.955282 30837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:19:02.954800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.60:35767 (local address 127.0.0.1:54454)
0620 20:19:02.955018 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0620 20:19:02.955114 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.217.60:35767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260620 20:19:02.956264 30837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:19:02.955941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.60:35767 (local address 127.0.0.1:54464)
0620 20:19:02.956079 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0620 20:19:02.956150 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.217.60:35767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260620 20:19:02.956499 30834 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.217.60:35767: Network error: Client connection negotiation failed: client connection to 127.16.217.60:35767: connect: Connection refused (error 111)
I20260620 20:19:02.976554 30760 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.217.61" port: 41137 } attempt: 5
Jun 20 20:19:02 dist-test-slave-2hpg krb5kdc[30680](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986742, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.217.62@KRBTEST.COM for kudu/127.16.217.61@KRBTEST.COM
I20260620 20:19:02.991927 30839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:19:02.977386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59488 (local address 127.16.217.61:41137)
0620 20:19:02.977566 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0620 20:19:02.977570 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:19:02.977595 (+ 25us) server_negotiation.cc:407] Connection header received
0620 20:19:02.977854 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:19:02.977859 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:19:02.978001 (+ 142us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:19:02.978108 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:19:02.979067 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:19:02.979809 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:19:02.981642 (+ 1833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:19:02.982086 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:19:02.982192 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:19:02.986233 (+ 4041us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:19:02.986238 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:19:02.986242 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:19:02.986275 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:19:02.988996 (+ 2721us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:19:02.989944 (+ 948us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:19:02.989949 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:19:02.989951 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:19:02.990071 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:19:02.990557 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:19:02.990560 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:19:02.990562 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:19:02.990842 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:19:02.990941 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0620 20:19:02.991494 (+ 553us) server_negotiation.cc:299] Negotiation successful
0620 20:19:02.991705 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 20:19:02.992224 30762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:19:02.977290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.61:41137 (local address 127.0.0.1:59488)
0620 20:19:02.977522 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0620 20:19:02.977545 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 20:19:02.977676 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:19:02.978166 (+ 490us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:19:02.978171 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:19:02.978666 (+ 495us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:19:02.978892 (+ 226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:19:02.978904 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:19:02.979955 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:19:02.979963 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:19:02.981467 (+ 1504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:19:02.981481 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:19:02.982246 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:19:02.982254 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:19:02.982498 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:19:02.983193 (+ 695us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:19:02.983223 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:19:02.986029 (+ 2806us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:19:02.989155 (+ 3126us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:19:02.989162 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:19:02.989169 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:19:02.989764 (+ 595us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:19:02.990213 (+ 449us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:19:02.990219 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:19:02.990225 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:19:02.990432 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:19:02.990985 (+ 553us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:19:02.990993 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:19:02.991221 (+ 228us) client_negotiation.cc:769] Sending connection context
0620 20:19:02.991408 (+ 187us) client_negotiation.cc:240] Negotiation successful
0620 20:19:02.991765 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":80}
I20260620 20:19:02.993266 30760 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.217.60" port: 35767 } has no permanent_uuid. Determining permanent_uuid...
I20260620 20:19:02.994370 30762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:19:02.993927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.60:35767 (local address 127.0.0.1:54478)
0620 20:19:02.994117 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0620 20:19:02.994209 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.217.60:35767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260620 20:19:02.995316 30762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:19:02.994944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.60:35767 (local address 127.0.0.1:54488)
0620 20:19:02.995103 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0620 20:19:02.995203 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.217.60:35767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260620 20:19:02.995534 30760 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.217.60:35767: Network error: Client connection negotiation failed: client connection to 127.16.217.60:35767: connect: Connection refused (error 111)
I20260620 20:19:03.046777 30760 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.217.60" port: 35767 } attempt: 1
I20260620 20:19:03.047735 30834 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.217.60" port: 35767 } attempt: 3
I20260620 20:19:03.048040 30762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:19:03.047479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.60:35767 (local address 127.0.0.1:54494)
0620 20:19:03.047721 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0620 20:19:03.047829 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.217.60:35767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
I20260620 20:19:03.048923 30837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:19:03.048323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.60:35767 (local address 127.0.0.1:54502)
0620 20:19:03.048608 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0620 20:19:03.048722 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.217.60:35767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122}
I20260620 20:19:03.049116 30762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:19:03.048723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.60:35767 (local address 127.0.0.1:54518)
0620 20:19:03.048872 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0620 20:19:03.048964 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.217.60:35767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260620 20:19:03.049392 30760 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.217.60:35767: Network error: Client connection negotiation failed: client connection to 127.16.217.60:35767: connect: Connection refused (error 111)
I20260620 20:19:03.050153 30837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:19:03.049633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.60:35767 (local address 127.0.0.1:54530)
0620 20:19:03.049920 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0620 20:19:03.050012 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.217.60:35767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134}
W20260620 20:19:03.050370 30834 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.217.60:35767: Network error: Client connection negotiation failed: client connection to 127.16.217.60:35767: connect: Connection refused (error 111)
I20260620 20:19:03.117640 30760 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.217.60" port: 35767 } attempt: 2
I20260620 20:19:03.118817 30762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:19:03.118333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.60:35767 (local address 127.0.0.1:54538)
0620 20:19:03.118532 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0620 20:19:03.118653 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.217.60:35767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260620 20:19:03.119867 30762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:19:03.119515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.60:35767 (local address 127.0.0.1:54548)
0620 20:19:03.119657 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0620 20:19:03.119732 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.217.60:35767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260620 20:19:03.120174 30760 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.217.60:35767: Network error: Client connection negotiation failed: client connection to 127.16.217.60:35767: connect: Connection refused (error 111)
W20260620 20:19:03.162508 30841 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:19:03.162796 30841 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:19:03.162963 30841 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:19:03.170037 30841 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:19:03.170136 30841 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:19:03.170249 30841 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:19:03.170349 30841 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:19:03.170435 30841 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:19:03.180073 30841 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.217.20:34695
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986536687152-17252-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986536687152-17252-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.16.217.62:45423,127.16.217.61:41137,127.16.217.60:35767
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.60.keytab
--principal=kudu/127.16.217.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.217.60:35767
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986536687152-17252-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.16.217.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986536687152-17252-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:19:03.181438 30841 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:19:03.182767 30841 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 20:19:03.193707 30834 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.217.60" port: 35767 } attempt: 4
I20260620 20:19:03.194967 30837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:19:03.194333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.60:35767 (local address 127.0.0.1:54560)
0620 20:19:03.194700 (+ 367us) negotiation.cc:107] Waiting for socket to connect
0620 20:19:03.194789 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.217.60:35767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":220}
I20260620 20:19:03.196262 30837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:19:03.195881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.60:35767 (local address 127.0.0.1:54564)
0620 20:19:03.196060 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0620 20:19:03.196147 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.217.60:35767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
W20260620 20:19:03.196820 30847 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:19:03.197438 30834 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.217.60:35767: Network error: Client connection negotiation failed: client connection to 127.16.217.60:35767: connect: Connection refused (error 111)
W20260620 20:19:03.198235 30846 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:19:03.199945 30849 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:19:03.201082 30841 server_base.cc:1034] running on GCE node
I20260620 20:19:03.202035 30841 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:19:03.203451 30841 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:19:03.204723 30841 hybrid_clock.cc:648] HybridClock initialized: now 1781986743204668 us; error 54 us; skew 500 ppm
I20260620 20:19:03.205137 30841 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:19:03 dist-test-slave-2hpg krb5kdc[30680](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986743, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.217.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:19:03.208413 30841 init.cc:377] Logged in from keytab as kudu/127.16.217.60@KRBTEST.COM (short username kudu)
I20260620 20:19:03.210137 30841 webserver.cc:460] Webserver started at http://127.16.217.60:35389/ using document root <none> and password file <none>
I20260620 20:19:03.210889 30841 fs_manager.cc:359] Metadata directory not provided
I20260620 20:19:03.211047 30841 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:19:03.211367 30841 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:19:03.214593 30841 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986536687152-17252-0/minicluster-data/master-2/data/instance:
uuid: "d68d89c7c018414a9dc4100d6867b977"
format_stamp: "Formatted at 2026-06-20 20:19:03 on dist-test-slave-2hpg"
server_key: "e3a87de7a0e9abe9a29eb6e7df5c6177"
server_key_iv: "72088d72ae1857687e6356b4ab3de80b"
server_key_version: "encryptionkey@0"
I20260620 20:19:03.215451 30841 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986536687152-17252-0/minicluster-data/master-2/wal/instance:
uuid: "d68d89c7c018414a9dc4100d6867b977"
format_stamp: "Formatted at 2026-06-20 20:19:03 on dist-test-slave-2hpg"
server_key: "e3a87de7a0e9abe9a29eb6e7df5c6177"
server_key_iv: "72088d72ae1857687e6356b4ab3de80b"
server_key_version: "encryptionkey@0"
I20260620 20:19:03.221112 30841 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.005s
I20260620 20:19:03.224722 30856 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:19:03.226194 30841 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 20:19:03.226413 30841 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986536687152-17252-0/minicluster-data/master-2/data,/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986536687152-17252-0/minicluster-data/master-2/wal
uuid: "d68d89c7c018414a9dc4100d6867b977"
format_stamp: "Formatted at 2026-06-20 20:19:03 on dist-test-slave-2hpg"
server_key: "e3a87de7a0e9abe9a29eb6e7df5c6177"
server_key_iv: "72088d72ae1857687e6356b4ab3de80b"
server_key_version: "encryptionkey@0"
I20260620 20:19:03.226478 30760 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.217.60" port: 35767 } attempt: 3
I20260620 20:19:03.226631 30841 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986536687152-17252-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986536687152-17252-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986536687152-17252-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:19:03.227880 30762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:19:03.227141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.60:35767 (local address 127.0.0.1:54574)
0620 20:19:03.227379 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0620 20:19:03.227523 (+ 144us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.217.60:35767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260620 20:19:03.229058 30762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:19:03.228672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.60:35767 (local address 127.0.0.1:54580)
0620 20:19:03.228831 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0620 20:19:03.228945 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.217.60:35767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260620 20:19:03.229382 30760 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.217.60:35767: Network error: Client connection negotiation failed: client connection to 127.16.217.60:35767: connect: Connection refused (error 111)
I20260620 20:19:03.246488 30841 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:19:03.250890 30841 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:19:03.251210 30841 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:19:03.270972 30841 rpc_server.cc:307] RPC server started. Bound to: 127.16.217.60:35767
I20260620 20:19:03.270989 30907 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.217.60:35767 every 8 connection(s)
I20260620 20:19:03.272765 30841 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986536687152-17252-0/minicluster-data/master-2/data/info.pb
I20260620 20:19:03.276629 30908 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:19:03.280903 17252 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu as pid 30841
I20260620 20:19:03.281098 17252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986536687152-17252-0/minicluster-data/master-2/wal/instance
I20260620 20:19:03.281565 17252 external_mini_cluster.cc:1428] Setting key c98257cd8ac381c388b49ccdf5764b5d
I20260620 20:19:03.283613 30908 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.217.62" port: 45423 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 20:19:03 dist-test-slave-2hpg krb5kdc[30680](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986743, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.217.60@KRBTEST.COM for kudu/127.16.217.62@KRBTEST.COM
Jun 20 20:19:03 dist-test-slave-2hpg krb5kdc[30680](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986741, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.217.60@KRBTEST.COM
I20260620 20:19:03.301775 30763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:19:03.286351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33434 (local address 127.16.217.62:45423)
0620 20:19:03.286556 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0620 20:19:03.286563 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:19:03.287202 (+ 639us) server_negotiation.cc:407] Connection header received
0620 20:19:03.287473 (+ 271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:19:03.287478 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:19:03.287550 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:19:03.287662 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:19:03.288931 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:19:03.289617 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:19:03.291158 (+ 1541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:19:03.291617 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:19:03.291756 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:19:03.295456 (+ 3700us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:19:03.295464 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:19:03.295471 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:19:03.295513 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:19:03.298347 (+ 2834us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:19:03.299220 (+ 873us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:19:03.299224 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:19:03.299226 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:19:03.299368 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:19:03.299871 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:19:03.299874 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:19:03.299876 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:19:03.300261 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:19:03.300437 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0620 20:19:03.301285 (+ 848us) server_negotiation.cc:299] Negotiation successful
0620 20:19:03.301505 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 20:19:03.303064 30912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:19:03.286286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:45423 (local address 127.0.0.1:33434)
0620 20:19:03.287067 (+ 781us) negotiation.cc:107] Waiting for socket to connect
0620 20:19:03.287115 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0620 20:19:03.287323 (+ 208us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:19:03.287721 (+ 398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:19:03.287728 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:19:03.288311 (+ 583us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:19:03.288771 (+ 460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:19:03.288787 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:19:03.289825 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:19:03.289828 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:19:03.290972 (+ 1144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:19:03.290983 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:19:03.291845 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:19:03.291853 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:19:03.292004 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:19:03.292732 (+ 728us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:19:03.292752 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:19:03.295193 (+ 2441us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:19:03.298486 (+ 3293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:19:03.298495 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:19:03.298510 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:19:03.299098 (+ 588us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:19:03.299485 (+ 387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:19:03.299492 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:19:03.299498 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:19:03.299761 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:19:03.300401 (+ 640us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:19:03.300407 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:19:03.300828 (+ 421us) client_negotiation.cc:769] Sending connection context
0620 20:19:03.301119 (+ 291us) client_negotiation.cc:240] Negotiation successful
0620 20:19:03.301661 (+ 542us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":560,"mutex_wait_us":2,"thread_start_us":421,"threads_started":1}
I20260620 20:19:03.303313 30911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:19:03.283452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54594 (local address 127.16.217.60:35767)
0620 20:19:03.284291 (+ 839us) server_negotiation.cc:206] Beginning negotiation
0620 20:19:03.284302 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 20:19:03.284341 (+ 39us) server_negotiation.cc:407] Connection header received
0620 20:19:03.286260 (+ 1919us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:19:03.286281 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:19:03.286721 (+ 440us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:19:03.287097 (+ 376us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:19:03.288530 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:19:03.289688 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:19:03.291032 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:19:03.291536 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:19:03.291756 (+ 220us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:19:03.296386 (+ 4630us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:19:03.296397 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:19:03.296416 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:19:03.296457 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:19:03.299839 (+ 3382us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:19:03.300662 (+ 823us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:19:03.300669 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:19:03.300672 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:19:03.300816 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:19:03.301330 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:19:03.301335 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:19:03.301338 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:19:03.301744 (+ 406us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:19:03.302032 (+ 288us) server_negotiation.cc:1035] Waiting for connection context
0620 20:19:03.302643 (+ 611us) server_negotiation.cc:299] Negotiation successful
0620 20:19:03.302884 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":627,"thread_start_us":425,"threads_started":1}
I20260620 20:19:03.304663 30908 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.217.61" port: 41137 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 20:19:03 dist-test-slave-2hpg krb5kdc[30680](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986743, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.217.60@KRBTEST.COM for kudu/127.16.217.61@KRBTEST.COM
Jun 20 20:19:03 dist-test-slave-2hpg krb5kdc[30680](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986741, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.217.62@KRBTEST.COM
I20260620 20:19:03.319730 30839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:19:03.305474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59492 (local address 127.16.217.61:41137)
0620 20:19:03.305656 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0620 20:19:03.305662 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:19:03.305783 (+ 121us) server_negotiation.cc:407] Connection header received
0620 20:19:03.305999 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:19:03.306009 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:19:03.306073 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:19:03.306176 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:19:03.306967 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:19:03.308131 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:19:03.309832 (+ 1701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:19:03.310508 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:19:03.310633 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:19:03.313809 (+ 3176us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:19:03.313817 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:19:03.313823 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:19:03.313902 (+ 79us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:19:03.317044 (+ 3142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:19:03.317904 (+ 860us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:19:03.317911 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:19:03.317916 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:19:03.318074 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:19:03.318530 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:19:03.318533 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:19:03.318535 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:19:03.318780 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:19:03.318886 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0620 20:19:03.319326 (+ 440us) server_negotiation.cc:299] Negotiation successful
0620 20:19:03.319474 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 20:19:03.319891 30912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:19:03.305222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.61:41137 (local address 127.0.0.1:59492)
0620 20:19:03.305691 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0620 20:19:03.305707 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 20:19:03.305836 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:19:03.306193 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:19:03.306197 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:19:03.306625 (+ 428us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:19:03.306811 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:19:03.306825 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:19:03.308342 (+ 1517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:19:03.308347 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:19:03.309700 (+ 1353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:19:03.309711 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:19:03.310648 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:19:03.310652 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:19:03.310801 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:19:03.311383 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:19:03.311399 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:19:03.313618 (+ 2219us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:19:03.317190 (+ 3572us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:19:03.317197 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:19:03.317202 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:19:03.317713 (+ 511us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:19:03.318208 (+ 495us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:19:03.318213 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:19:03.318217 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:19:03.318417 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:19:03.318915 (+ 498us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:19:03.318919 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:19:03.319088 (+ 169us) client_negotiation.cc:769] Sending connection context
0620 20:19:03.319250 (+ 162us) client_negotiation.cc:240] Negotiation successful
0620 20:19:03.319533 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":343}
I20260620 20:19:03.321007 30908 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.217.60" port: 35767 } has no permanent_uuid. Determining permanent_uuid...
I20260620 20:19:03.323218 30763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:19:03.306397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33440 (local address 127.16.217.62:45423)
0620 20:19:03.306560 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0620 20:19:03.306567 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:19:03.306599 (+ 32us) server_negotiation.cc:407] Connection header received
0620 20:19:03.306692 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:19:03.306697 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:19:03.306770 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:19:03.306869 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:19:03.308365 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:19:03.309456 (+ 1091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:19:03.311119 (+ 1663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:19:03.311625 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:19:03.311716 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:19:03.316213 (+ 4497us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:19:03.316218 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:19:03.316222 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:19:03.316251 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:19:03.319712 (+ 3461us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:19:03.320857 (+ 1145us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:19:03.320862 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:19:03.320865 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:19:03.321027 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:19:03.321477 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:19:03.321480 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:19:03.321483 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:19:03.322091 (+ 608us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:19:03.322393 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0620 20:19:03.322670 (+ 277us) server_negotiation.cc:299] Negotiation successful
0620 20:19:03.322898 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
Jun 20 20:19:03 dist-test-slave-2hpg krb5kdc[30680](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986743, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.217.60@KRBTEST.COM for kudu/127.16.217.60@KRBTEST.COM
Jun 20 20:19:03 dist-test-slave-2hpg krb5kdc[30680](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986741, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.217.61@KRBTEST.COM
I20260620 20:19:03.335286 30911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:19:03.321858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54606 (local address 127.16.217.60:35767)
0620 20:19:03.322036 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0620 20:19:03.322041 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:19:03.322108 (+ 67us) server_negotiation.cc:407] Connection header received
0620 20:19:03.322261 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:19:03.322265 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:19:03.322321 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:19:03.322414 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:19:03.323105 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:19:03.324009 (+ 904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:19:03.325349 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:19:03.326066 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:19:03.326208 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:19:03.329512 (+ 3304us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:19:03.329518 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:19:03.329522 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:19:03.329570 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:19:03.332409 (+ 2839us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:19:03.333188 (+ 779us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:19:03.333197 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:19:03.333200 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:19:03.333343 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:19:03.333802 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:19:03.333810 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:19:03.333814 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:19:03.334139 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:19:03.334284 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0620 20:19:03.334677 (+ 393us) server_negotiation.cc:299] Negotiation successful
0620 20:19:03.334993 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":111,"server-negotiator.queue_time_us":37}
I20260620 20:19:03.335263 30912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:19:03.321764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.60:35767 (local address 127.0.0.1:54606)
0620 20:19:03.322029 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0620 20:19:03.322044 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 20:19:03.322160 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:19:03.322427 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:19:03.322431 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:19:03.322798 (+ 367us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:19:03.322926 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:19:03.322934 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:19:03.324156 (+ 1222us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:19:03.324161 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:19:03.325215 (+ 1054us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:19:03.325227 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:19:03.326239 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:19:03.326244 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:19:03.326409 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:19:03.327027 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:19:03.327049 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:19:03.329321 (+ 2272us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:19:03.332538 (+ 3217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:19:03.332545 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:19:03.332549 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:19:03.333056 (+ 507us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:19:03.333506 (+ 450us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:19:03.333513 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:19:03.333519 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:19:03.333663 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:19:03.334261 (+ 598us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:19:03.334266 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:19:03.334405 (+ 139us) client_negotiation.cc:769] Sending connection context
0620 20:19:03.334548 (+ 143us) client_negotiation.cc:240] Negotiation successful
0620 20:19:03.334852 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160}
I20260620 20:19:03.340180 30839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:19:03.324831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59508 (local address 127.16.217.61:41137)
0620 20:19:03.324981 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0620 20:19:03.324988 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:19:03.325006 (+ 18us) server_negotiation.cc:407] Connection header received
0620 20:19:03.325153 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:19:03.325159 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:19:03.325225 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:19:03.325324 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:19:03.326853 (+ 1529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:19:03.327645 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:19:03.328957 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:19:03.329369 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:19:03.329515 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:19:03.334414 (+ 4899us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:19:03.334419 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:19:03.334422 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:19:03.334451 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:19:03.337736 (+ 3285us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:19:03.338410 (+ 674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:19:03.338417 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:19:03.338421 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:19:03.338586 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:19:03.338916 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:19:03.338920 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:19:03.338923 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:19:03.339241 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:19:03.339517 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0620 20:19:03.339645 (+ 128us) server_negotiation.cc:299] Negotiation successful
0620 20:19:03.339873 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 20:19:03.341329 30908 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d68d89c7c018414a9dc4100d6867b977: Bootstrap starting.
I20260620 20:19:03.344916 17252 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 20:19:03.345230 30908 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d68d89c7c018414a9dc4100d6867b977: Neither blocks nor log segments found. Creating new log.
I20260620 20:19:03.346489 30908 log.cc:824] T 00000000000000000000000000000000 P d68d89c7c018414a9dc4100d6867b977: Log is configured to *not* fsync() on all Append() calls
I20260620 20:19:03.349701 30908 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d68d89c7c018414a9dc4100d6867b977: No bootstrap required, opened a new log
I20260620 20:19:03.360889 30908 raft_consensus.cc:348] T 00000000000000000000000000000000 P d68d89c7c018414a9dc4100d6867b977 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e27d0330bb6a43258dcba9fe7cf90eb1" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 45423 } } peers { permanent_uuid: "861816685d044c968d3fd1013f1fff55" member_type: VOTER last_known_addr { host: "127.16.217.61" port: 41137 } } peers { permanent_uuid: "d68d89c7c018414a9dc4100d6867b977" member_type: VOTER last_known_addr { host: "127.16.217.60" port: 35767 } }
I20260620 20:19:03.361361 30908 raft_consensus.cc:374] T 00000000000000000000000000000000 P d68d89c7c018414a9dc4100d6867b977 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:19:03.361480 30908 raft_consensus.cc:729] T 00000000000000000000000000000000 P d68d89c7c018414a9dc4100d6867b977 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d68d89c7c018414a9dc4100d6867b977, State: Initialized, Role: FOLLOWER
I20260620 20:19:03.362260 30908 consensus_queue.cc:260] T 00000000000000000000000000000000 P d68d89c7c018414a9dc4100d6867b977 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e27d0330bb6a43258dcba9fe7cf90eb1" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 45423 } } peers { permanent_uuid: "861816685d044c968d3fd1013f1fff55" member_type: VOTER last_known_addr { host: "127.16.217.61" port: 41137 } } peers { permanent_uuid: "d68d89c7c018414a9dc4100d6867b977" member_type: VOTER last_known_addr { host: "127.16.217.60" port: 35767 } }
I20260620 20:19:03.371323 30763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:19:03.350683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33456 (local address 127.16.217.62:45423)
0620 20:19:03.350919 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0620 20:19:03.350924 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:19:03.351257 (+ 333us) server_negotiation.cc:407] Connection header received
0620 20:19:03.351714 (+ 457us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:19:03.351720 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:19:03.351799 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:19:03.351868 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:19:03.354113 (+ 2245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:19:03.355220 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:19:03.357263 (+ 2043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:19:03.357955 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:19:03.358068 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:19:03.362006 (+ 3938us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:19:03.362013 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:19:03.362017 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:19:03.362065 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:19:03.366987 (+ 4922us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:19:03.368221 (+ 1234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:19:03.368230 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:19:03.368235 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:19:03.368435 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:19:03.369006 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:19:03.369013 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:19:03.369019 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:19:03.369450 (+ 431us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:19:03.369744 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0620 20:19:03.370471 (+ 727us) server_negotiation.cc:299] Negotiation successful
0620 20:19:03.370825 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 20:19:03.372674 30839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:19:03.351329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59524 (local address 127.16.217.61:41137)
0620 20:19:03.351490 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0620 20:19:03.351497 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:19:03.351962 (+ 465us) server_negotiation.cc:407] Connection header received
0620 20:19:03.352237 (+ 275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:19:03.352242 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:19:03.352313 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:19:03.352382 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:19:03.355299 (+ 2917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:19:03.356671 (+ 1372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:19:03.359464 (+ 2793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:19:03.360355 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:19:03.360471 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:19:03.364333 (+ 3862us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:19:03.364339 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:19:03.364344 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:19:03.364382 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:19:03.367824 (+ 3442us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:19:03.370027 (+ 2203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:19:03.370032 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:19:03.370036 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:19:03.370213 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:19:03.370715 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:19:03.370720 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:19:03.370725 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:19:03.371152 (+ 427us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:19:03.371387 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0620 20:19:03.372025 (+ 638us) server_negotiation.cc:299] Negotiation successful
0620 20:19:03.372286 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260620 20:19:03.378558 30922 sys_catalog.cc:455] T 00000000000000000000000000000000 P d68d89c7c018414a9dc4100d6867b977 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e27d0330bb6a43258dcba9fe7cf90eb1" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 45423 } } peers { permanent_uuid: "861816685d044c968d3fd1013f1fff55" member_type: VOTER last_known_addr { host: "127.16.217.61" port: 41137 } } peers { permanent_uuid: "d68d89c7c018414a9dc4100d6867b977" member_type: VOTER last_known_addr { host: "127.16.217.60" port: 35767 } } }
I20260620 20:19:03.379073 30922 sys_catalog.cc:458] T 00000000000000000000000000000000 P d68d89c7c018414a9dc4100d6867b977 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 20:19:03.379325 30922 raft_consensus.cc:482] T 00000000000000000000000000000000 P d68d89c7c018414a9dc4100d6867b977 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 20:19:03.379678 30922 raft_consensus.cc:504] T 00000000000000000000000000000000 P d68d89c7c018414a9dc4100d6867b977 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e27d0330bb6a43258dcba9fe7cf90eb1" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 45423 } } peers { permanent_uuid: "861816685d044c968d3fd1013f1fff55" member_type: VOTER last_known_addr { host: "127.16.217.61" port: 41137 } } peers { permanent_uuid: "d68d89c7c018414a9dc4100d6867b977" member_type: VOTER last_known_addr { host: "127.16.217.60" port: 35767 } }
I20260620 20:19:03.380784 30911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:19:03.352290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54614 (local address 127.16.217.60:35767)
0620 20:19:03.352470 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0620 20:19:03.352478 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 20:19:03.363543 (+ 11065us) server_negotiation.cc:407] Connection header received
0620 20:19:03.363976 (+ 433us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:19:03.363983 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:19:03.364062 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:19:03.364134 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:19:03.365890 (+ 1756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:19:03.367052 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:19:03.369202 (+ 2150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:19:03.369943 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:19:03.370032 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:19:03.373830 (+ 3798us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:19:03.373836 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:19:03.373839 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:19:03.373903 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:19:03.376920 (+ 3017us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:19:03.377967 (+ 1047us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:19:03.377975 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:19:03.377984 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:19:03.378245 (+ 261us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:19:03.378782 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:19:03.378787 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:19:03.378793 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:19:03.379232 (+ 439us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:19:03.379628 (+ 396us) server_negotiation.cc:1035] Waiting for connection context
0620 20:19:03.379967 (+ 339us) server_negotiation.cc:299] Negotiation successful
0620 20:19:03.380208 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 20:19:03.382189 30922 leader_election.cc:288] T 00000000000000000000000000000000 P d68d89c7c018414a9dc4100d6867b977 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e27d0330bb6a43258dcba9fe7cf90eb1 (127.16.217.62:45423), 861816685d044c968d3fd1013f1fff55 (127.16.217.61:41137)
I20260620 20:19:03.383855 30908 sys_catalog.cc:564] T 00000000000000000000000000000000 P d68d89c7c018414a9dc4100d6867b977 [sys.catalog]: configured and running, proceeding with master startup.
W20260620 20:19:03.399574 30933 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d68d89c7c018414a9dc4100d6867b977: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 20:19:03.399710 30933 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 20:19:03.394984 30841 master_runner.cc:186] Error getting master registration for 127.16.217.62:45423: OK, instance_id { permanent_uuid: "e27d0330bb6a43258dcba9fe7cf90eb1" instance_seqno: 1781986742260412 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 20:19:03.406391 30841 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 20:19:03.406562 30760 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.217.60" port: 35767 } attempt: 4
Jun 20 20:19:03 dist-test-slave-2hpg krb5kdc[30680](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986742, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.217.62@KRBTEST.COM for kudu/127.16.217.60@KRBTEST.COM
I20260620 20:19:03.423130 30762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:19:03.407095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.60:35767 (local address 127.0.0.1:54628)
0620 20:19:03.407286 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0620 20:19:03.407305 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 20:19:03.407410 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:19:03.407715 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:19:03.407719 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:19:03.408224 (+ 505us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:19:03.408429 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:19:03.408442 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:19:03.409956 (+ 1514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:19:03.409963 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:19:03.411462 (+ 1499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:19:03.411476 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:19:03.412457 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:19:03.412465 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:19:03.412676 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:19:03.413361 (+ 685us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:19:03.413385 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:19:03.416231 (+ 2846us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:19:03.420184 (+ 3953us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:19:03.420189 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:19:03.420194 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:19:03.420748 (+ 554us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:19:03.421240 (+ 492us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:19:03.421247 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:19:03.421253 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:19:03.421449 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:19:03.422076 (+ 627us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:19:03.422080 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:19:03.422271 (+ 191us) client_negotiation.cc:769] Sending connection context
0620 20:19:03.422427 (+ 156us) client_negotiation.cc:240] Negotiation successful
0620 20:19:03.422720 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":65}
I20260620 20:19:03.423591 30911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:19:03.407218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54628 (local address 127.16.217.60:35767)
0620 20:19:03.407383 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0620 20:19:03.407393 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 20:19:03.407410 (+ 17us) server_negotiation.cc:407] Connection header received
0620 20:19:03.407513 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:19:03.407518 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:19:03.407588 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:19:03.407693 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:19:03.408615 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:19:03.409791 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:19:03.411601 (+ 1810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:19:03.412300 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:19:03.412425 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:19:03.416415 (+ 3990us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:19:03.416423 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:19:03.416428 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:19:03.416465 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:19:03.420017 (+ 3552us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:19:03.420936 (+ 919us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:19:03.420943 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:19:03.420949 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:19:03.421117 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:19:03.421560 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:19:03.421566 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:19:03.421570 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:19:03.421956 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:19:03.422119 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0620 20:19:03.422528 (+ 409us) server_negotiation.cc:299] Negotiation successful
0620 20:19:03.422847 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 20:19:03.429766 30760 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e27d0330bb6a43258dcba9fe7cf90eb1: Bootstrap starting.
I20260620 20:19:03.434399 30760 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e27d0330bb6a43258dcba9fe7cf90eb1: Neither blocks nor log segments found. Creating new log.
I20260620 20:19:03.435714 30760 log.cc:824] T 00000000000000000000000000000000 P e27d0330bb6a43258dcba9fe7cf90eb1: Log is configured to *not* fsync() on all Append() calls
I20260620 20:19:03.439060 30760 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e27d0330bb6a43258dcba9fe7cf90eb1: No bootstrap required, opened a new log
I20260620 20:19:03.450343 30760 raft_consensus.cc:348] T 00000000000000000000000000000000 P e27d0330bb6a43258dcba9fe7cf90eb1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e27d0330bb6a43258dcba9fe7cf90eb1" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 45423 } } peers { permanent_uuid: "861816685d044c968d3fd1013f1fff55" member_type: VOTER last_known_addr { host: "127.16.217.61" port: 41137 } } peers { permanent_uuid: "d68d89c7c018414a9dc4100d6867b977" member_type: VOTER last_known_addr { host: "127.16.217.60" port: 35767 } }
I20260620 20:19:03.450755 30760 raft_consensus.cc:374] T 00000000000000000000000000000000 P e27d0330bb6a43258dcba9fe7cf90eb1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:19:03.450874 30760 raft_consensus.cc:729] T 00000000000000000000000000000000 P e27d0330bb6a43258dcba9fe7cf90eb1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e27d0330bb6a43258dcba9fe7cf90eb1, State: Initialized, Role: FOLLOWER
I20260620 20:19:03.451540 30760 consensus_queue.cc:260] T 00000000000000000000000000000000 P e27d0330bb6a43258dcba9fe7cf90eb1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e27d0330bb6a43258dcba9fe7cf90eb1" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 45423 } } peers { permanent_uuid: "861816685d044c968d3fd1013f1fff55" member_type: VOTER last_known_addr { host: "127.16.217.61" port: 41137 } } peers { permanent_uuid: "d68d89c7c018414a9dc4100d6867b977" member_type: VOTER last_known_addr { host: "127.16.217.60" port: 35767 } }
I20260620 20:19:03.452541 30936 sys_catalog.cc:455] T 00000000000000000000000000000000 P e27d0330bb6a43258dcba9fe7cf90eb1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e27d0330bb6a43258dcba9fe7cf90eb1" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 45423 } } peers { permanent_uuid: "861816685d044c968d3fd1013f1fff55" member_type: VOTER last_known_addr { host: "127.16.217.61" port: 41137 } } peers { permanent_uuid: "d68d89c7c018414a9dc4100d6867b977" member_type: VOTER last_known_addr { host: "127.16.217.60" port: 35767 } } }
I20260620 20:19:03.452999 30936 sys_catalog.cc:458] T 00000000000000000000000000000000 P e27d0330bb6a43258dcba9fe7cf90eb1 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 20:19:03.453859 30760 sys_catalog.cc:564] T 00000000000000000000000000000000 P e27d0330bb6a43258dcba9fe7cf90eb1 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:19:03.453950 30735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d68d89c7c018414a9dc4100d6867b977" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e27d0330bb6a43258dcba9fe7cf90eb1" is_pre_election: true
I20260620 20:19:03.454555 30735 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e27d0330bb6a43258dcba9fe7cf90eb1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d68d89c7c018414a9dc4100d6867b977 in term 0.
I20260620 20:19:03.455196 30857 leader_election.cc:302] T 00000000000000000000000000000000 P d68d89c7c018414a9dc4100d6867b977 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d68d89c7c018414a9dc4100d6867b977, e27d0330bb6a43258dcba9fe7cf90eb1; no voters:
I20260620 20:19:03.455752 30922 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d68d89c7c018414a9dc4100d6867b977 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 20:19:03.455924 30922 raft_consensus.cc:482] T 00000000000000000000000000000000 P d68d89c7c018414a9dc4100d6867b977 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 20:19:03.456027 30922 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d68d89c7c018414a9dc4100d6867b977 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:19:03.458930 30922 raft_consensus.cc:504] T 00000000000000000000000000000000 P d68d89c7c018414a9dc4100d6867b977 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e27d0330bb6a43258dcba9fe7cf90eb1" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 45423 } } peers { permanent_uuid: "861816685d044c968d3fd1013f1fff55" member_type: VOTER last_known_addr { host: "127.16.217.61" port: 41137 } } peers { permanent_uuid: "d68d89c7c018414a9dc4100d6867b977" member_type: VOTER last_known_addr { host: "127.16.217.60" port: 35767 } }
I20260620 20:19:03.460078 30735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d68d89c7c018414a9dc4100d6867b977" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e27d0330bb6a43258dcba9fe7cf90eb1"
I20260620 20:19:03.460284 30735 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e27d0330bb6a43258dcba9fe7cf90eb1 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:19:03.462962 30735 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e27d0330bb6a43258dcba9fe7cf90eb1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d68d89c7c018414a9dc4100d6867b977 in term 1.
I20260620 20:19:03.463378 30922 leader_election.cc:288] T 00000000000000000000000000000000 P d68d89c7c018414a9dc4100d6867b977 [CANDIDATE]: Term 1 election: Requested vote from peers e27d0330bb6a43258dcba9fe7cf90eb1 (127.16.217.62:45423), 861816685d044c968d3fd1013f1fff55 (127.16.217.61:41137)
I20260620 20:19:03.463411 30857 leader_election.cc:302] T 00000000000000000000000000000000 P d68d89c7c018414a9dc4100d6867b977 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d68d89c7c018414a9dc4100d6867b977, e27d0330bb6a43258dcba9fe7cf90eb1; no voters:
I20260620 20:19:03.463721 30921 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d68d89c7c018414a9dc4100d6867b977 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:19:03.465507 30921 raft_consensus.cc:686] T 00000000000000000000000000000000 P d68d89c7c018414a9dc4100d6867b977 [term 1 LEADER]: Becoming Leader. State: Replica: d68d89c7c018414a9dc4100d6867b977, State: Running, Role: LEADER
I20260620 20:19:03.466197 30921 consensus_queue.cc:237] T 00000000000000000000000000000000 P d68d89c7c018414a9dc4100d6867b977 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e27d0330bb6a43258dcba9fe7cf90eb1" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 45423 } } peers { permanent_uuid: "861816685d044c968d3fd1013f1fff55" member_type: VOTER last_known_addr { host: "127.16.217.61" port: 41137 } } peers { permanent_uuid: "d68d89c7c018414a9dc4100d6867b977" member_type: VOTER last_known_addr { host: "127.16.217.60" port: 35767 } }
I20260620 20:19:03.470001 30922 sys_catalog.cc:455] T 00000000000000000000000000000000 P d68d89c7c018414a9dc4100d6867b977 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d68d89c7c018414a9dc4100d6867b977. Latest consensus state: current_term: 1 leader_uuid: "d68d89c7c018414a9dc4100d6867b977" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e27d0330bb6a43258dcba9fe7cf90eb1" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 45423 } } peers { permanent_uuid: "861816685d044c968d3fd1013f1fff55" member_type: VOTER last_known_addr { host: "127.16.217.61" port: 41137 } } peers { permanent_uuid: "d68d89c7c018414a9dc4100d6867b977" member_type: VOTER last_known_addr { host: "127.16.217.60" port: 35767 } } }
I20260620 20:19:03.470361 30922 sys_catalog.cc:458] T 00000000000000000000000000000000 P d68d89c7c018414a9dc4100d6867b977 [sys.catalog]: This master's current role is: LEADER
I20260620 20:19:03.472775 30944 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 20:19:03.476377 30944 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 20:19:03.490267 30735 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e27d0330bb6a43258dcba9fe7cf90eb1 [term 1 FOLLOWER]: Refusing update from remote peer d68d89c7c018414a9dc4100d6867b977: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:19:03.491719 30922 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d68d89c7c018414a9dc4100d6867b977 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e27d0330bb6a43258dcba9fe7cf90eb1" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 45423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:19:03.496636 30834 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.217.60" port: 35767 } attempt: 5
W20260620 20:19:03.498456 30714 tablet.cc:2369] T 00000000000000000000000000000000 P e27d0330bb6a43258dcba9fe7cf90eb1: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 20:19:03.503350 30862 tablet.cc:2369] T 00000000000000000000000000000000 P d68d89c7c018414a9dc4100d6867b977: Can't schedule compaction. Clean time has not been advanced past its initial value.
Jun 20 20:19:03 dist-test-slave-2hpg krb5kdc[30680](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986742, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.217.61@KRBTEST.COM for kudu/127.16.217.60@KRBTEST.COM
I20260620 20:19:03.519520 30936 sys_catalog.cc:455] T 00000000000000000000000000000000 P e27d0330bb6a43258dcba9fe7cf90eb1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d68d89c7c018414a9dc4100d6867b977. Latest consensus state: current_term: 1 leader_uuid: "d68d89c7c018414a9dc4100d6867b977" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e27d0330bb6a43258dcba9fe7cf90eb1" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 45423 } } peers { permanent_uuid: "861816685d044c968d3fd1013f1fff55" member_type: VOTER last_known_addr { host: "127.16.217.61" port: 41137 } } peers { permanent_uuid: "d68d89c7c018414a9dc4100d6867b977" member_type: VOTER last_known_addr { host: "127.16.217.60" port: 35767 } } }
I20260620 20:19:03.519906 30936 sys_catalog.cc:458] T 00000000000000000000000000000000 P e27d0330bb6a43258dcba9fe7cf90eb1 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 20:19:03.523521 30946 mvcc.cc:205] Tried to move back new op lower bound from 7299017701316571136 to 7299017701242662912. Current Snapshot: MvccSnapshot[applied={T|T < 7299017701316571136}]
I20260620 20:19:03.526537 30911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:19:03.497860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54632 (local address 127.16.217.60:35767)
0620 20:19:03.498055 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0620 20:19:03.498061 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:19:03.498078 (+ 17us) server_negotiation.cc:407] Connection header received
0620 20:19:03.498130 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:19:03.498135 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:19:03.498206 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:19:03.498297 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:19:03.499509 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:19:03.500665 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:19:03.502642 (+ 1977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:19:03.503322 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:19:03.503448 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:19:03.511625 (+ 8177us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:19:03.511637 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:19:03.511644 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:19:03.511685 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:19:03.515577 (+ 3892us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:19:03.517016 (+ 1439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:19:03.517053 (+ 37us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:19:03.517058 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:19:03.517349 (+ 291us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:19:03.518197 (+ 848us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:19:03.518203 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:19:03.518207 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:19:03.518956 (+ 749us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:19:03.519214 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0620 20:19:03.520412 (+ 1198us) server_negotiation.cc:299] Negotiation successful
0620 20:19:03.522448 (+ 2036us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 20:19:03.527371 30837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:19:03.497198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.60:35767 (local address 127.0.0.1:54632)
0620 20:19:03.497526 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0620 20:19:03.497547 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 20:19:03.497669 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:19:03.498482 (+ 813us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:19:03.498487 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:19:03.499055 (+ 568us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:19:03.499292 (+ 237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:19:03.499307 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:19:03.500953 (+ 1646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:19:03.500959 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:19:03.502456 (+ 1497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:19:03.502474 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:19:03.506735 (+ 4261us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:19:03.506741 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:19:03.506960 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:19:03.507880 (+ 920us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:19:03.507905 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:19:03.511259 (+ 3354us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:19:03.515852 (+ 4593us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:19:03.515857 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:19:03.515862 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:19:03.516815 (+ 953us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:19:03.517663 (+ 848us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:19:03.517669 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:19:03.517674 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:19:03.517988 (+ 314us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:19:03.519371 (+ 1383us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:19:03.519376 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:19:03.519680 (+ 304us) client_negotiation.cc:769] Sending connection context
0620 20:19:03.520884 (+ 1204us) client_negotiation.cc:240] Negotiation successful
0620 20:19:03.523261 (+ 2377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":204}
I20260620 20:19:03.529369 30921 sys_catalog.cc:455] T 00000000000000000000000000000000 P d68d89c7c018414a9dc4100d6867b977 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d68d89c7c018414a9dc4100d6867b977" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e27d0330bb6a43258dcba9fe7cf90eb1" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 45423 } } peers { permanent_uuid: "861816685d044c968d3fd1013f1fff55" member_type: VOTER last_known_addr { host: "127.16.217.61" port: 41137 } } peers { permanent_uuid: "d68d89c7c018414a9dc4100d6867b977" member_type: VOTER last_known_addr { host: "127.16.217.60" port: 35767 } } }
I20260620 20:19:03.529794 30921 sys_catalog.cc:458] T 00000000000000000000000000000000 P d68d89c7c018414a9dc4100d6867b977 [sys.catalog]: This master's current role is: LEADER
I20260620 20:19:03.534161 30834 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 861816685d044c968d3fd1013f1fff55: Bootstrap starting.
I20260620 20:19:03.539143 30834 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 861816685d044c968d3fd1013f1fff55: Neither blocks nor log segments found. Creating new log.
I20260620 20:19:03.544418 30834 log.cc:824] T 00000000000000000000000000000000 P 861816685d044c968d3fd1013f1fff55: Log is configured to *not* fsync() on all Append() calls
I20260620 20:19:03.547482 30944 catalog_manager.cc:1347] Generated new cluster ID: 16b7cbac2b22484f9457e7b13920ddac
I20260620 20:19:03.547596 30944 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 20:19:03.569445 30834 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 861816685d044c968d3fd1013f1fff55: No bootstrap required, opened a new log
I20260620 20:19:03.580642 30936 sys_catalog.cc:455] T 00000000000000000000000000000000 P e27d0330bb6a43258dcba9fe7cf90eb1 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d68d89c7c018414a9dc4100d6867b977" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e27d0330bb6a43258dcba9fe7cf90eb1" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 45423 } } peers { permanent_uuid: "861816685d044c968d3fd1013f1fff55" member_type: VOTER last_known_addr { host: "127.16.217.61" port: 41137 } } peers { permanent_uuid: "d68d89c7c018414a9dc4100d6867b977" member_type: VOTER last_known_addr { host: "127.16.217.60" port: 35767 } } }
I20260620 20:19:03.580921 30936 sys_catalog.cc:458] T 00000000000000000000000000000000 P e27d0330bb6a43258dcba9fe7cf90eb1 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 20:19:03.581555 30834 raft_consensus.cc:348] T 00000000000000000000000000000000 P 861816685d044c968d3fd1013f1fff55 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e27d0330bb6a43258dcba9fe7cf90eb1" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 45423 } } peers { permanent_uuid: "861816685d044c968d3fd1013f1fff55" member_type: VOTER last_known_addr { host: "127.16.217.61" port: 41137 } } peers { permanent_uuid: "d68d89c7c018414a9dc4100d6867b977" member_type: VOTER last_known_addr { host: "127.16.217.60" port: 35767 } }
I20260620 20:19:03.582063 30834 raft_consensus.cc:374] T 00000000000000000000000000000000 P 861816685d044c968d3fd1013f1fff55 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:19:03.582243 30834 raft_consensus.cc:729] T 00000000000000000000000000000000 P 861816685d044c968d3fd1013f1fff55 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 861816685d044c968d3fd1013f1fff55, State: Initialized, Role: FOLLOWER
I20260620 20:19:03.583107 30944 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 20:19:03.582979 30834 consensus_queue.cc:260] T 00000000000000000000000000000000 P 861816685d044c968d3fd1013f1fff55 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e27d0330bb6a43258dcba9fe7cf90eb1" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 45423 } } peers { permanent_uuid: "861816685d044c968d3fd1013f1fff55" member_type: VOTER last_known_addr { host: "127.16.217.61" port: 41137 } } peers { permanent_uuid: "d68d89c7c018414a9dc4100d6867b977" member_type: VOTER last_known_addr { host: "127.16.217.60" port: 35767 } }
I20260620 20:19:03.585011 30944 catalog_manager.cc:1504] Loading token signing keys...
I20260620 20:19:03.587152 30693 master_runner.cc:186] Error getting master registration for 127.16.217.61:41137: OK, instance_id { permanent_uuid: "861816685d044c968d3fd1013f1fff55" instance_seqno: 1781986742764784 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 20:19:03.597325 30693 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260620 20:19:03.608717 30953 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e27d0330bb6a43258dcba9fe7cf90eb1: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 20:19:03.608949 30953 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 20:19:03.610738 30944 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d68d89c7c018414a9dc4100d6867b977: Generated new TSK 0
I20260620 20:19:03.612190 30944 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 20:19:03.620790 30958 sys_catalog.cc:455] T 00000000000000000000000000000000 P 861816685d044c968d3fd1013f1fff55 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e27d0330bb6a43258dcba9fe7cf90eb1" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 45423 } } peers { permanent_uuid: "861816685d044c968d3fd1013f1fff55" member_type: VOTER last_known_addr { host: "127.16.217.61" port: 41137 } } peers { permanent_uuid: "d68d89c7c018414a9dc4100d6867b977" member_type: VOTER last_known_addr { host: "127.16.217.60" port: 35767 } } }
I20260620 20:19:03.621265 30958 sys_catalog.cc:458] T 00000000000000000000000000000000 P 861816685d044c968d3fd1013f1fff55 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 20:19:03.622367 30834 sys_catalog.cc:564] T 00000000000000000000000000000000 P 861816685d044c968d3fd1013f1fff55 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:19:03.622763 30807 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 861816685d044c968d3fd1013f1fff55 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:19:03.628106 30807 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 861816685d044c968d3fd1013f1fff55 [term 1 FOLLOWER]: Refusing update from remote peer d68d89c7c018414a9dc4100d6867b977: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:19:03.622995 30809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d68d89c7c018414a9dc4100d6867b977" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "861816685d044c968d3fd1013f1fff55" is_pre_election: true
I20260620 20:19:03.630689 30921 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d68d89c7c018414a9dc4100d6867b977 [LEADER]: Connected to new peer: Peer: permanent_uuid: "861816685d044c968d3fd1013f1fff55" member_type: VOTER last_known_addr { host: "127.16.217.61" port: 41137 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:19:03.631383 30808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d68d89c7c018414a9dc4100d6867b977" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "861816685d044c968d3fd1013f1fff55"
I20260620 20:19:03.635885 30808 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 861816685d044c968d3fd1013f1fff55 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d68d89c7c018414a9dc4100d6867b977 in term 1.
I20260620 20:19:03.703426 30958 sys_catalog.cc:455] T 00000000000000000000000000000000 P 861816685d044c968d3fd1013f1fff55 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d68d89c7c018414a9dc4100d6867b977. Latest consensus state: current_term: 1 leader_uuid: "d68d89c7c018414a9dc4100d6867b977" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e27d0330bb6a43258dcba9fe7cf90eb1" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 45423 } } peers { permanent_uuid: "861816685d044c968d3fd1013f1fff55" member_type: VOTER last_known_addr { host: "127.16.217.61" port: 41137 } } peers { permanent_uuid: "d68d89c7c018414a9dc4100d6867b977" member_type: VOTER last_known_addr { host: "127.16.217.60" port: 35767 } } }
I20260620 20:19:03.703761 30958 sys_catalog.cc:458] T 00000000000000000000000000000000 P 861816685d044c968d3fd1013f1fff55 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 20:19:03.704166 30972 sys_catalog.cc:455] T 00000000000000000000000000000000 P 861816685d044c968d3fd1013f1fff55 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d68d89c7c018414a9dc4100d6867b977" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e27d0330bb6a43258dcba9fe7cf90eb1" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 45423 } } peers { permanent_uuid: "861816685d044c968d3fd1013f1fff55" member_type: VOTER last_known_addr { host: "127.16.217.61" port: 41137 } } peers { permanent_uuid: "d68d89c7c018414a9dc4100d6867b977" member_type: VOTER last_known_addr { host: "127.16.217.60" port: 35767 } } }
I20260620 20:19:03.704484 30972 sys_catalog.cc:458] T 00000000000000000000000000000000 P 861816685d044c968d3fd1013f1fff55 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 20:19:03.713347 30978 catalog_manager.cc:1259] Loaded cluster ID: 16b7cbac2b22484f9457e7b13920ddac
I20260620 20:19:03.713510 30978 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 861816685d044c968d3fd1013f1fff55: loading cluster ID for follower catalog manager: success
I20260620 20:19:03.717377 30978 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 861816685d044c968d3fd1013f1fff55: acquiring CA information for follower catalog manager: success
I20260620 20:19:03.719872 30978 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 861816685d044c968d3fd1013f1fff55: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 20:19:04.172561 30921 sys_catalog.cc:455] T 00000000000000000000000000000000 P d68d89c7c018414a9dc4100d6867b977 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d68d89c7c018414a9dc4100d6867b977" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e27d0330bb6a43258dcba9fe7cf90eb1" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 45423 } } peers { permanent_uuid: "861816685d044c968d3fd1013f1fff55" member_type: VOTER last_known_addr { host: "127.16.217.61" port: 41137 } } peers { permanent_uuid: "d68d89c7c018414a9dc4100d6867b977" member_type: VOTER last_known_addr { host: "127.16.217.60" port: 35767 } } }
I20260620 20:19:04.173190 30921 sys_catalog.cc:458] T 00000000000000000000000000000000 P d68d89c7c018414a9dc4100d6867b977 [sys.catalog]: This master's current role is: LEADER
I20260620 20:19:04.611121 30953 catalog_manager.cc:1259] Loaded cluster ID: 16b7cbac2b22484f9457e7b13920ddac
I20260620 20:19:04.611220 30953 catalog_manager.cc:1552] T 00000000000000000000000000000000 P e27d0330bb6a43258dcba9fe7cf90eb1: loading cluster ID for follower catalog manager: success
I20260620 20:19:04.613742 30953 catalog_manager.cc:1574] T 00000000000000000000000000000000 P e27d0330bb6a43258dcba9fe7cf90eb1: acquiring CA information for follower catalog manager: success
I20260620 20:19:04.615581 30953 catalog_manager.cc:1602] T 00000000000000000000000000000000 P e27d0330bb6a43258dcba9fe7cf90eb1: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 20:19:04.743697 17252 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu pid 30693
I20260620 20:19:04.744979 30755 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:19:04.855029 17252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu with pid 30693
I20260620 20:19:04.875661 17252 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu pid 30767
I20260620 20:19:04.876686 30829 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:19:04.983054 17252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu with pid 30767
I20260620 20:19:04.995663 30985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:19:04.994932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:45423 (local address 127.0.0.1:33470)
0620 20:19:04.995379 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0620 20:19:04.995489 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.217.62:45423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":244,"threads_started":1}
W20260620 20:19:04.995947 30857 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.62:45423: connect: Connection refused (error 111)
I20260620 20:19:04.998966 30985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:19:04.998615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:45423 (local address 127.0.0.1:33472)
0620 20:19:04.998775 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0620 20:19:04.998848 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.217.62:45423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260620 20:19:04.999156 30857 consensus_peers.cc:487] T 00000000000000000000000000000000 P d68d89c7c018414a9dc4100d6867b977 -> Peer e27d0330bb6a43258dcba9fe7cf90eb1 (127.16.217.62:45423): Couldn't send request to peer e27d0330bb6a43258dcba9fe7cf90eb1. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.62:45423: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260620 20:19:05.004415 17252 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu pid 30841
I20260620 20:19:05.005532 30903 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:19:05.114521 17252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu with pid 30841
2026-06-20T20:19:05Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (4653 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781986536687152-17252-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 20:19:05 dist-test-slave-2hpg krb5kdc[30990](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 20:19:05 dist-test-slave-2hpg krb5kdc[30990](info): set up 2 sockets
Jun 20 20:19:05 dist-test-slave-2hpg krb5kdc[30990](info): commencing operation
krb5kdc: starting...
W20260620 20:19:06.222776 17252 mini_kdc.cc:121] Time spent starting KDC: real 1.066s user 0.007s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781986536687152-17252-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781986536687152-17252-0/krb5kdc/test-user.keytab.
Jun 20 20:19:06 dist-test-slave-2hpg krb5kdc[30990](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986746, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T20:19:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T20:19:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.217.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.217.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.217.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.62.keytab.
Entry for principal kudu/127.16.217.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.62.keytab.
I20260620 20:19:06.457293 17252 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
/tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781986536687152-17252-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781986536687152-17252-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781986536687152-17252-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781986536687152-17252-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.217.62:43657
--webserver_interface=127.16.217.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.62.keytab
--principal=kudu/127.16.217.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.217.20:40627
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.217.62:43657
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781986536687152-17252-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781986536687152-17252-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781986536687152-17252-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:19:06.788537 31003 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:19:06.788851 31003 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:19:06.789057 31003 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:19:06.796458 31003 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:19:06.796629 31003 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:19:06.796729 31003 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:19:06.796840 31003 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:19:06.796936 31003 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:19:06.807713 31003 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.217.20:40627
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781986536687152-17252-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781986536687152-17252-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.217.62:43657
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.62.keytab
--principal=kudu/127.16.217.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.217.62:43657
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781986536687152-17252-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.217.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781986536687152-17252-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:19:06.809139 31003 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:19:06.810642 31003 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:19:06.823443 31008 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:19:06.823398 31011 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:19:06.823488 31009 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:19:06.824748 31003 server_base.cc:1034] running on GCE node
I20260620 20:19:06.825524 31003 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:19:06.827040 31003 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:19:06.828265 31003 hybrid_clock.cc:648] HybridClock initialized: now 1781986746828212 us; error 40 us; skew 500 ppm
I20260620 20:19:06.828680 31003 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:19:06 dist-test-slave-2hpg krb5kdc[30990](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986746, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.217.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:19:06.831894 31003 init.cc:377] Logged in from keytab as kudu/127.16.217.62@KRBTEST.COM (short username kudu)
I20260620 20:19:06.833514 31003 webserver.cc:460] Webserver started at http://127.16.217.62:40239/ using document root <none> and password file <none>
I20260620 20:19:06.834314 31003 fs_manager.cc:359] Metadata directory not provided
I20260620 20:19:06.834472 31003 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:19:06.834816 31003 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:19:06.838073 31003 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781986536687152-17252-0/minicluster-data/master-0/data/instance:
uuid: "654bb9bb1eef405dad69855108ecf182"
format_stamp: "Formatted at 2026-06-20 20:19:06 on dist-test-slave-2hpg"
server_key: "fe5163dbf01a28cf045e127f8d2bd336"
server_key_iv: "daccb73d060bfc79b2aa3833b5e9960e"
server_key_version: "encryptionkey@0"
I20260620 20:19:06.838914 31003 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781986536687152-17252-0/minicluster-data/master-0/wal/instance:
uuid: "654bb9bb1eef405dad69855108ecf182"
format_stamp: "Formatted at 2026-06-20 20:19:06 on dist-test-slave-2hpg"
server_key: "fe5163dbf01a28cf045e127f8d2bd336"
server_key_iv: "daccb73d060bfc79b2aa3833b5e9960e"
server_key_version: "encryptionkey@0"
I20260620 20:19:06.844450 31003 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.005s
I20260620 20:19:06.848248 31018 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:19:06.849700 31003 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260620 20:19:06.849851 31003 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781986536687152-17252-0/minicluster-data/master-0/data,/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781986536687152-17252-0/minicluster-data/master-0/wal
uuid: "654bb9bb1eef405dad69855108ecf182"
format_stamp: "Formatted at 2026-06-20 20:19:06 on dist-test-slave-2hpg"
server_key: "fe5163dbf01a28cf045e127f8d2bd336"
server_key_iv: "daccb73d060bfc79b2aa3833b5e9960e"
server_key_version: "encryptionkey@0"
I20260620 20:19:06.850096 31003 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781986536687152-17252-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781986536687152-17252-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781986536687152-17252-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:19:06.879489 31003 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:19:06.884187 31003 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:19:06.884516 31003 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:19:06.903861 31003 rpc_server.cc:307] RPC server started. Bound to: 127.16.217.62:43657
I20260620 20:19:06.903883 31069 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.217.62:43657 every 8 connection(s)
I20260620 20:19:06.905261 31003 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781986536687152-17252-0/minicluster-data/master-0/data/info.pb
I20260620 20:19:06.909420 31070 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:19:06.910046 17252 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu as pid 31003
I20260620 20:19:06.910290 17252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781986536687152-17252-0/minicluster-data/master-0/wal/instance
I20260620 20:19:06.910944 17252 external_mini_cluster.cc:1428] Setting key d47b49f1da3002e52e743855a701f91c
I20260620 20:19:06.924057 31070 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 654bb9bb1eef405dad69855108ecf182: Bootstrap starting.
Jun 20 20:19:06 dist-test-slave-2hpg krb5kdc[30990](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986746, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.217.62@KRBTEST.COM
I20260620 20:19:06.927315 31070 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 654bb9bb1eef405dad69855108ecf182: Neither blocks nor log segments found. Creating new log.
I20260620 20:19:06.928655 31070 log.cc:824] T 00000000000000000000000000000000 P 654bb9bb1eef405dad69855108ecf182: Log is configured to *not* fsync() on all Append() calls
I20260620 20:19:06.932312 31070 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 654bb9bb1eef405dad69855108ecf182: No bootstrap required, opened a new log
I20260620 20:19:06.936712 31073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:19:06.912871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51972 (local address 127.16.217.62:43657)
0620 20:19:06.913666 (+ 795us) server_negotiation.cc:206] Beginning negotiation
0620 20:19:06.913681 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 20:19:06.913735 (+ 54us) server_negotiation.cc:407] Connection header received
0620 20:19:06.916332 (+ 2597us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:19:06.916361 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:19:06.916915 (+ 554us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:19:06.917369 (+ 454us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:19:06.918730 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:19:06.920801 (+ 2071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:19:06.922244 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:19:06.923272 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:19:06.923449 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:19:06.927775 (+ 4326us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:19:06.927792 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:19:06.927812 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:19:06.927878 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:19:06.932079 (+ 4201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:19:06.932848 (+ 769us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:19:06.932856 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:19:06.932865 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:19:06.933074 (+ 209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:19:06.933452 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:19:06.933459 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:19:06.933462 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:19:06.934216 (+ 754us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:19:06.934540 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0620 20:19:06.934880 (+ 340us) server_negotiation.cc:299] Negotiation successful
0620 20:19:06.935344 (+ 464us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":539,"thread_start_us":310,"threads_started":1}
I20260620 20:19:06.940593 17252 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 20:19:06.942205 31070 raft_consensus.cc:348] T 00000000000000000000000000000000 P 654bb9bb1eef405dad69855108ecf182 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "654bb9bb1eef405dad69855108ecf182" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 43657 } }
I20260620 20:19:06.942555 31070 raft_consensus.cc:374] T 00000000000000000000000000000000 P 654bb9bb1eef405dad69855108ecf182 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:19:06.942651 31070 raft_consensus.cc:729] T 00000000000000000000000000000000 P 654bb9bb1eef405dad69855108ecf182 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 654bb9bb1eef405dad69855108ecf182, State: Initialized, Role: FOLLOWER
I20260620 20:19:06.943274 31070 consensus_queue.cc:260] T 00000000000000000000000000000000 P 654bb9bb1eef405dad69855108ecf182 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "654bb9bb1eef405dad69855108ecf182" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 43657 } }
I20260620 20:19:06.943516 31070 raft_consensus.cc:388] T 00000000000000000000000000000000 P 654bb9bb1eef405dad69855108ecf182 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 20:19:06.943661 31070 raft_consensus.cc:482] T 00000000000000000000000000000000 P 654bb9bb1eef405dad69855108ecf182 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 20:19:06.943841 31070 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 654bb9bb1eef405dad69855108ecf182 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:19:06.946005 31070 raft_consensus.cc:504] T 00000000000000000000000000000000 P 654bb9bb1eef405dad69855108ecf182 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "654bb9bb1eef405dad69855108ecf182" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 43657 } }
I20260620 20:19:06.946504 31070 leader_election.cc:302] T 00000000000000000000000000000000 P 654bb9bb1eef405dad69855108ecf182 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 654bb9bb1eef405dad69855108ecf182; no voters:
I20260620 20:19:06.947065 31070 leader_election.cc:288] T 00000000000000000000000000000000 P 654bb9bb1eef405dad69855108ecf182 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 20:19:06.947204 31075 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 654bb9bb1eef405dad69855108ecf182 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:19:06.947938 31075 raft_consensus.cc:686] T 00000000000000000000000000000000 P 654bb9bb1eef405dad69855108ecf182 [term 1 LEADER]: Becoming Leader. State: Replica: 654bb9bb1eef405dad69855108ecf182, State: Running, Role: LEADER
I20260620 20:19:06.948523 31075 consensus_queue.cc:237] T 00000000000000000000000000000000 P 654bb9bb1eef405dad69855108ecf182 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "654bb9bb1eef405dad69855108ecf182" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 43657 } }
I20260620 20:19:06.949096 31070 sys_catalog.cc:564] T 00000000000000000000000000000000 P 654bb9bb1eef405dad69855108ecf182 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:19:06.951047 31076 sys_catalog.cc:455] T 00000000000000000000000000000000 P 654bb9bb1eef405dad69855108ecf182 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "654bb9bb1eef405dad69855108ecf182" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "654bb9bb1eef405dad69855108ecf182" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 43657 } } }
I20260620 20:19:06.951380 31076 sys_catalog.cc:458] T 00000000000000000000000000000000 P 654bb9bb1eef405dad69855108ecf182 [sys.catalog]: This master's current role is: LEADER
I20260620 20:19:06.951596 31077 sys_catalog.cc:455] T 00000000000000000000000000000000 P 654bb9bb1eef405dad69855108ecf182 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 654bb9bb1eef405dad69855108ecf182. Latest consensus state: current_term: 1 leader_uuid: "654bb9bb1eef405dad69855108ecf182" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "654bb9bb1eef405dad69855108ecf182" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 43657 } } }
I20260620 20:19:06.951988 31077 sys_catalog.cc:458] T 00000000000000000000000000000000 P 654bb9bb1eef405dad69855108ecf182 [sys.catalog]: This master's current role is: LEADER
I20260620 20:19:06.954571 31081 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 20:19:06.960429 31081 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 20:19:06.969578 31081 catalog_manager.cc:1347] Generated new cluster ID: 5edf2cb8341d42e8a399e870b9f25f70
I20260620 20:19:06.969738 31081 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 20:19:06.986328 31081 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 20:19:06.987582 31081 catalog_manager.cc:1504] Loading token signing keys...
I20260620 20:19:06.999850 31081 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 654bb9bb1eef405dad69855108ecf182: Generated new TSK 0
I20260620 20:19:07.000669 31081 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 20:19:07.053138 17252 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu pid 31003
I20260620 20:19:07.054402 31065 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:19:07.152108 17252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu with pid 31003
2026-06-20T20:19:07Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (2033 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986536687152-17252-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 20:19:07 dist-test-slave-2hpg krb5kdc[31098](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 20:19:07 dist-test-slave-2hpg krb5kdc[31098](info): set up 2 sockets
Jun 20 20:19:07 dist-test-slave-2hpg krb5kdc[31098](info): commencing operation
krb5kdc: starting...
W20260620 20:19:08.314534 17252 mini_kdc.cc:121] Time spent starting KDC: real 1.123s user 0.004s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986536687152-17252-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986536687152-17252-0/krb5kdc/test-user.keytab.
Jun 20 20:19:08 dist-test-slave-2hpg krb5kdc[31098](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986748, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T20:19:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T20:19:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.217.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.217.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.217.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.62.keytab.
Entry for principal kudu/127.16.217.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.62.keytab.
I20260620 20:19:08.550369 17252 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
/tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986536687152-17252-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986536687152-17252-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986536687152-17252-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986536687152-17252-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.217.62:36477
--webserver_interface=127.16.217.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.62.keytab
--principal=kudu/127.16.217.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.217.20:38487
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.217.62:36477
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986536687152-17252-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986536687152-17252-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986536687152-17252-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:19:08.883599 31111 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:19:08.884133 31111 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:19:08.884251 31111 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:19:08.892179 31111 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:19:08.892289 31111 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:19:08.892336 31111 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:19:08.892395 31111 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:19:08.892439 31111 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:19:08.903679 31111 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.217.20:38487
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986536687152-17252-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986536687152-17252-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.217.62:36477
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.62.keytab
--principal=kudu/127.16.217.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.217.62:36477
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986536687152-17252-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.217.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986536687152-17252-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:19:08.904991 31111 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:19:08.906430 31111 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:19:08.918898 31119 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:19:08.918989 31117 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:19:08.918962 31116 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:19:08.920135 31111 server_base.cc:1034] running on GCE node
I20260620 20:19:08.920861 31111 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:19:08.922241 31111 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:19:08.923453 31111 hybrid_clock.cc:648] HybridClock initialized: now 1781986748923387 us; error 54 us; skew 500 ppm
I20260620 20:19:08.923815 31111 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:19:08 dist-test-slave-2hpg krb5kdc[31098](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986748, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.217.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:19:08.927093 31111 init.cc:377] Logged in from keytab as kudu/127.16.217.62@KRBTEST.COM (short username kudu)
I20260620 20:19:08.929208 31111 webserver.cc:460] Webserver started at http://127.16.217.62:36691/ using document root <none> and password file <none>
I20260620 20:19:08.930159 31111 fs_manager.cc:359] Metadata directory not provided
I20260620 20:19:08.930313 31111 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:19:08.930671 31111 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:19:08.934877 31111 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986536687152-17252-0/minicluster-data/master-0/data/instance:
uuid: "becbda298fcf43ab9c61810d35e9764a"
format_stamp: "Formatted at 2026-06-20 20:19:08 on dist-test-slave-2hpg"
server_key: "c6d40976d637a0ffe5a5a7c91c0ba6c9"
server_key_iv: "7208137db4a722e0cbcf6f8b467ef330"
server_key_version: "encryptionkey@0"
I20260620 20:19:08.935798 31111 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986536687152-17252-0/minicluster-data/master-0/wal/instance:
uuid: "becbda298fcf43ab9c61810d35e9764a"
format_stamp: "Formatted at 2026-06-20 20:19:08 on dist-test-slave-2hpg"
server_key: "c6d40976d637a0ffe5a5a7c91c0ba6c9"
server_key_iv: "7208137db4a722e0cbcf6f8b467ef330"
server_key_version: "encryptionkey@0"
I20260620 20:19:08.941069 31111 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 20:19:08.944561 31126 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:19:08.945979 31111 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 20:19:08.946164 31111 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986536687152-17252-0/minicluster-data/master-0/data,/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986536687152-17252-0/minicluster-data/master-0/wal
uuid: "becbda298fcf43ab9c61810d35e9764a"
format_stamp: "Formatted at 2026-06-20 20:19:08 on dist-test-slave-2hpg"
server_key: "c6d40976d637a0ffe5a5a7c91c0ba6c9"
server_key_iv: "7208137db4a722e0cbcf6f8b467ef330"
server_key_version: "encryptionkey@0"
I20260620 20:19:08.946362 31111 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986536687152-17252-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986536687152-17252-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986536687152-17252-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:19:08.961422 31111 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:19:08.966369 31111 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:19:08.966681 31111 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:19:08.986970 31111 rpc_server.cc:307] RPC server started. Bound to: 127.16.217.62:36477
I20260620 20:19:08.987016 31177 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.217.62:36477 every 8 connection(s)
I20260620 20:19:08.988471 31111 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986536687152-17252-0/minicluster-data/master-0/data/info.pb
I20260620 20:19:08.992309 17252 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu as pid 31111
I20260620 20:19:08.992326 31178 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:19:08.992511 17252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986536687152-17252-0/minicluster-data/master-0/wal/instance
I20260620 20:19:08.993170 17252 external_mini_cluster.cc:1428] Setting key ecfe235cfc1d8ad5cf8f8de336218ce3
I20260620 20:19:09.006799 31178 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P becbda298fcf43ab9c61810d35e9764a: Bootstrap starting.
Jun 20 20:19:09 dist-test-slave-2hpg krb5kdc[31098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986748, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.217.62@KRBTEST.COM
I20260620 20:19:09.010398 31178 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P becbda298fcf43ab9c61810d35e9764a: Neither blocks nor log segments found. Creating new log.
I20260620 20:19:09.011822 31178 log.cc:824] T 00000000000000000000000000000000 P becbda298fcf43ab9c61810d35e9764a: Log is configured to *not* fsync() on all Append() calls
I20260620 20:19:09.015669 31178 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P becbda298fcf43ab9c61810d35e9764a: No bootstrap required, opened a new log
I20260620 20:19:09.019537 31181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:19:08.995228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56318 (local address 127.16.217.62:36477)
0620 20:19:08.996119 (+ 891us) server_negotiation.cc:206] Beginning negotiation
0620 20:19:08.996134 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 20:19:08.996184 (+ 50us) server_negotiation.cc:407] Connection header received
0620 20:19:08.999131 (+ 2947us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:19:08.999159 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:19:08.999674 (+ 515us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:19:09.000112 (+ 438us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:19:09.001437 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:19:09.003477 (+ 2040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:19:09.004856 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:19:09.005956 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:19:09.006135 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:19:09.010564 (+ 4429us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:19:09.010576 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:19:09.010597 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:19:09.010655 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:19:09.014710 (+ 4055us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:19:09.015574 (+ 864us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:19:09.015583 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:19:09.015591 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:19:09.015818 (+ 227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:19:09.016224 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:19:09.016230 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:19:09.016233 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:19:09.016953 (+ 720us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:19:09.017281 (+ 328us) server_negotiation.cc:1035] Waiting for connection context
0620 20:19:09.017584 (+ 303us) server_negotiation.cc:299] Negotiation successful
0620 20:19:09.018110 (+ 526us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":621,"thread_start_us":331,"threads_started":1}
I20260620 20:19:09.025626 31178 raft_consensus.cc:348] T 00000000000000000000000000000000 P becbda298fcf43ab9c61810d35e9764a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "becbda298fcf43ab9c61810d35e9764a" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 36477 } }
I20260620 20:19:09.026132 31178 raft_consensus.cc:374] T 00000000000000000000000000000000 P becbda298fcf43ab9c61810d35e9764a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:19:09.026253 31178 raft_consensus.cc:729] T 00000000000000000000000000000000 P becbda298fcf43ab9c61810d35e9764a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: becbda298fcf43ab9c61810d35e9764a, State: Initialized, Role: FOLLOWER
I20260620 20:19:09.026968 31178 consensus_queue.cc:260] T 00000000000000000000000000000000 P becbda298fcf43ab9c61810d35e9764a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "becbda298fcf43ab9c61810d35e9764a" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 36477 } }
I20260620 20:19:09.027268 31178 raft_consensus.cc:388] T 00000000000000000000000000000000 P becbda298fcf43ab9c61810d35e9764a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 20:19:09.027437 31178 raft_consensus.cc:482] T 00000000000000000000000000000000 P becbda298fcf43ab9c61810d35e9764a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 20:19:09.027652 31178 raft_consensus.cc:3037] T 00000000000000000000000000000000 P becbda298fcf43ab9c61810d35e9764a [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:19:09.030134 31178 raft_consensus.cc:504] T 00000000000000000000000000000000 P becbda298fcf43ab9c61810d35e9764a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "becbda298fcf43ab9c61810d35e9764a" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 36477 } }
I20260620 20:19:09.030705 31178 leader_election.cc:302] T 00000000000000000000000000000000 P becbda298fcf43ab9c61810d35e9764a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: becbda298fcf43ab9c61810d35e9764a; no voters:
I20260620 20:19:09.031445 31178 leader_election.cc:288] T 00000000000000000000000000000000 P becbda298fcf43ab9c61810d35e9764a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 20:19:09.031592 31183 raft_consensus.cc:2781] T 00000000000000000000000000000000 P becbda298fcf43ab9c61810d35e9764a [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:19:09.032328 31183 raft_consensus.cc:686] T 00000000000000000000000000000000 P becbda298fcf43ab9c61810d35e9764a [term 1 LEADER]: Becoming Leader. State: Replica: becbda298fcf43ab9c61810d35e9764a, State: Running, Role: LEADER
I20260620 20:19:09.032855 31183 consensus_queue.cc:237] T 00000000000000000000000000000000 P becbda298fcf43ab9c61810d35e9764a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "becbda298fcf43ab9c61810d35e9764a" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 36477 } }
I20260620 20:19:09.033643 31178 sys_catalog.cc:564] T 00000000000000000000000000000000 P becbda298fcf43ab9c61810d35e9764a [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:19:09.036055 31185 sys_catalog.cc:455] T 00000000000000000000000000000000 P becbda298fcf43ab9c61810d35e9764a [sys.catalog]: SysCatalogTable state changed. Reason: New leader becbda298fcf43ab9c61810d35e9764a. Latest consensus state: current_term: 1 leader_uuid: "becbda298fcf43ab9c61810d35e9764a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "becbda298fcf43ab9c61810d35e9764a" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 36477 } } }
I20260620 20:19:09.036417 31185 sys_catalog.cc:458] T 00000000000000000000000000000000 P becbda298fcf43ab9c61810d35e9764a [sys.catalog]: This master's current role is: LEADER
I20260620 20:19:09.037333 31190 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 20:19:09.037117 31184 sys_catalog.cc:455] T 00000000000000000000000000000000 P becbda298fcf43ab9c61810d35e9764a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "becbda298fcf43ab9c61810d35e9764a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "becbda298fcf43ab9c61810d35e9764a" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 36477 } } }
I20260620 20:19:09.037479 31184 sys_catalog.cc:458] T 00000000000000000000000000000000 P becbda298fcf43ab9c61810d35e9764a [sys.catalog]: This master's current role is: LEADER
I20260620 20:19:09.043497 31190 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 20:19:09.045359 31200 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260620 20:19:09.054447 31190 catalog_manager.cc:1347] Generated new cluster ID: a30997bc2e0d4a0b86613e6b28572b3a
I20260620 20:19:09.054605 31190 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 20:19:09.067438 31190 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 20:19:09.069397 31190 catalog_manager.cc:1504] Loading token signing keys...
I20260620 20:19:09.080951 31190 catalog_manager.cc:5868] T 00000000000000000000000000000000 P becbda298fcf43ab9c61810d35e9764a: Generated new TSK 0
I20260620 20:19:09.082011 31190 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.217.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.217.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.217.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.1.keytab.
Entry for principal kudu/127.16.217.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.1.keytab.
I20260620 20:19:09.155958 17252 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
/tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986536687152-17252-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986536687152-17252-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986536687152-17252-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986536687152-17252-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.217.1:0
--local_ip_for_outbound_sockets=127.16.217.1
--webserver_interface=127.16.217.1
--webserver_port=0
--tserver_master_addrs=127.16.217.62:36477
--keytab_file=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.1.keytab
--principal=kudu/127.16.217.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.217.20:38487
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986536687152-17252-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986536687152-17252-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986536687152-17252-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:19:09.498641 31204 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:19:09.498920 31204 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:19:09.499075 31204 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:19:09.506481 31204 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:19:09.506587 31204 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:19:09.506768 31204 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.217.1
I20260620 20:19:09.516774 31204 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.217.20:38487
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986536687152-17252-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986536687152-17252-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.1.keytab
--principal=kudu/127.16.217.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.217.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986536687152-17252-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.217.1
--webserver_port=0
--tserver_master_addrs=127.16.217.62:36477
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.217.1
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986536687152-17252-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:19:09.518154 31204 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:19:09.519536 31204 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:19:09.533063 31209 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:19:09.533116 31212 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:19:09.533179 31210 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:19:09.534055 31204 server_base.cc:1034] running on GCE node
I20260620 20:19:09.534684 31204 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:19:09.535676 31204 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:19:09.536931 31204 hybrid_clock.cc:648] HybridClock initialized: now 1781986749536882 us; error 42 us; skew 500 ppm
I20260620 20:19:09.537274 31204 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:19:09 dist-test-slave-2hpg krb5kdc[31098](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986749, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.217.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:19:09.540936 31204 init.cc:377] Logged in from keytab as kudu/127.16.217.1@KRBTEST.COM (short username kudu)
I20260620 20:19:09.543327 31204 webserver.cc:460] Webserver started at http://127.16.217.1:38787/ using document root <none> and password file <none>
I20260620 20:19:09.544080 31204 fs_manager.cc:359] Metadata directory not provided
I20260620 20:19:09.544188 31204 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:19:09.544422 31204 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:19:09.547447 31204 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986536687152-17252-0/minicluster-data/ts-0/data/instance:
uuid: "8400677523bb4227b8e2d7b83e7f45ba"
format_stamp: "Formatted at 2026-06-20 20:19:09 on dist-test-slave-2hpg"
server_key: "1282b3fbdc54571751d10a364a758a0f"
server_key_iv: "b0843de0908b3105cefbbeef1b335a8a"
server_key_version: "encryptionkey@0"
I20260620 20:19:09.548161 31204 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986536687152-17252-0/minicluster-data/ts-0/wal/instance:
uuid: "8400677523bb4227b8e2d7b83e7f45ba"
format_stamp: "Formatted at 2026-06-20 20:19:09 on dist-test-slave-2hpg"
server_key: "1282b3fbdc54571751d10a364a758a0f"
server_key_iv: "b0843de0908b3105cefbbeef1b335a8a"
server_key_version: "encryptionkey@0"
I20260620 20:19:09.553256 31204 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260620 20:19:09.556718 31219 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:19:09.558084 31204 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260620 20:19:09.558265 31204 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986536687152-17252-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986536687152-17252-0/minicluster-data/ts-0/wal
uuid: "8400677523bb4227b8e2d7b83e7f45ba"
format_stamp: "Formatted at 2026-06-20 20:19:09 on dist-test-slave-2hpg"
server_key: "1282b3fbdc54571751d10a364a758a0f"
server_key_iv: "b0843de0908b3105cefbbeef1b335a8a"
server_key_version: "encryptionkey@0"
I20260620 20:19:09.558480 31204 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986536687152-17252-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986536687152-17252-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986536687152-17252-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:19:09.572539 31204 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:19:09.576841 31204 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:19:09.577126 31204 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:19:09.578101 31204 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:19:09.579376 31204 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:19:09.579493 31204 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:19:09.579643 31204 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:19:09.579722 31204 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:19:09.613158 31204 rpc_server.cc:307] RPC server started. Bound to: 127.16.217.1:44135
I20260620 20:19:09.613206 31331 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.217.1:44135 every 8 connection(s)
I20260620 20:19:09.614504 31204 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986536687152-17252-0/minicluster-data/ts-0/data/info.pb
I20260620 20:19:09.622260 17252 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu as pid 31204
I20260620 20:19:09.622491 17252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986536687152-17252-0/minicluster-data/ts-0/wal/instance
I20260620 20:19:09.623152 17252 external_mini_cluster.cc:1428] Setting key 38a899d1f67e7d3d7bfb201c605fa025
Jun 20 20:19:09 dist-test-slave-2hpg krb5kdc[31098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986749, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.217.1@KRBTEST.COM for kudu/127.16.217.62@KRBTEST.COM
I20260620 20:19:09.636047 31334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:19:09.617128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.217.1:44117 (local address 127.16.217.62:36477)
0620 20:19:09.617574 (+ 446us) server_negotiation.cc:206] Beginning negotiation
0620 20:19:09.617579 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:19:09.618582 (+ 1003us) server_negotiation.cc:407] Connection header received
0620 20:19:09.620424 (+ 1842us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:19:09.620430 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:19:09.620529 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:19:09.620670 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:19:09.622441 (+ 1771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:19:09.623298 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:19:09.625280 (+ 1982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:19:09.625721 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:19:09.625829 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:19:09.629956 (+ 4127us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:19:09.629961 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:19:09.629964 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:19:09.629994 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:19:09.632688 (+ 2694us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:19:09.633425 (+ 737us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:19:09.633435 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:19:09.633437 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:19:09.633552 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:19:09.634094 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:19:09.634101 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:19:09.634123 (+ 22us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:19:09.634391 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:19:09.634554 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0620 20:19:09.635432 (+ 878us) server_negotiation.cc:299] Negotiation successful
0620 20:19:09.635674 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":328,"thread_start_us":218,"threads_started":1}
I20260620 20:19:09.637044 31335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:19:09.617675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:36477 (local address 127.16.217.1:44117)
0620 20:19:09.618408 (+ 733us) negotiation.cc:107] Waiting for socket to connect
0620 20:19:09.618450 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 20:19:09.620135 (+ 1685us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:19:09.620904 (+ 769us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:19:09.620912 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:19:09.621424 (+ 512us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:19:09.622223 (+ 799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:19:09.622253 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:19:09.623445 (+ 1192us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:19:09.623451 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:19:09.625114 (+ 1663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:19:09.625128 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:19:09.625985 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:19:09.625994 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:19:09.626322 (+ 328us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:19:09.627061 (+ 739us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:19:09.627094 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:19:09.629668 (+ 2574us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:19:09.632838 (+ 3170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:19:09.632848 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:19:09.632864 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:19:09.633280 (+ 416us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:19:09.633683 (+ 403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:19:09.633697 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:19:09.633700 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:19:09.633960 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:19:09.634524 (+ 564us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:19:09.634532 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:19:09.634970 (+ 438us) client_negotiation.cc:769] Sending connection context
0620 20:19:09.635314 (+ 344us) client_negotiation.cc:240] Negotiation successful
0620 20:19:09.635910 (+ 596us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":503,"thread_start_us":309,"threads_started":1}
I20260620 20:19:09.638756 31332 heartbeater.cc:347] Connected to a master server at 127.16.217.62:36477
I20260620 20:19:09.639104 31332 heartbeater.cc:464] Registering TS with master...
I20260620 20:19:09.639978 31332 heartbeater.cc:511] Master 127.16.217.62:36477 requested a full tablet report, sending...
I20260620 20:19:09.642483 31143 ts_manager.cc:194] Registered new tserver with Master: 8400677523bb4227b8e2d7b83e7f45ba (127.16.217.1:44135)
I20260620 20:19:09.644855 31143 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.217.1@KRBTEST.COM'} at 127.16.217.1:44117
WARNING: no policy specified for kudu/127.16.217.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.217.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.217.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.2.keytab.
Entry for principal kudu/127.16.217.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.2.keytab.
I20260620 20:19:09.676748 17252 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
/tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986536687152-17252-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986536687152-17252-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986536687152-17252-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986536687152-17252-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.217.2:0
--local_ip_for_outbound_sockets=127.16.217.2
--webserver_interface=127.16.217.2
--webserver_port=0
--tserver_master_addrs=127.16.217.62:36477
--keytab_file=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.2.keytab
--principal=kudu/127.16.217.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.217.20:38487
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986536687152-17252-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986536687152-17252-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986536687152-17252-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:19:10.011019 31338 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:19:10.011375 31338 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:19:10.011550 31338 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:19:10.019121 31338 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:19:10.019292 31338 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:19:10.019474 31338 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.217.2
I20260620 20:19:10.029520 31338 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.217.20:38487
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986536687152-17252-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986536687152-17252-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.2.keytab
--principal=kudu/127.16.217.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.217.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986536687152-17252-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.217.2
--webserver_port=0
--tserver_master_addrs=127.16.217.62:36477
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.217.2
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986536687152-17252-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:19:10.030947 31338 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:19:10.032303 31338 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:19:10.045795 31346 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:19:10.045928 31344 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:19:10.046268 31338 server_base.cc:1034] running on GCE node
W20260620 20:19:10.045944 31343 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:19:10.047209 31338 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:19:10.048113 31338 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:19:10.049351 31338 hybrid_clock.cc:648] HybridClock initialized: now 1781986750049317 us; error 44 us; skew 500 ppm
I20260620 20:19:10.049678 31338 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:19:10 dist-test-slave-2hpg krb5kdc[31098](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986750, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.217.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:19:10.052886 31338 init.cc:377] Logged in from keytab as kudu/127.16.217.2@KRBTEST.COM (short username kudu)
I20260620 20:19:10.054329 31338 webserver.cc:460] Webserver started at http://127.16.217.2:44817/ using document root <none> and password file <none>
I20260620 20:19:10.054934 31338 fs_manager.cc:359] Metadata directory not provided
I20260620 20:19:10.055029 31338 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:19:10.055280 31338 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:19:10.058214 31338 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986536687152-17252-0/minicluster-data/ts-1/data/instance:
uuid: "89eac4104e6c4573b766d41f6999395e"
format_stamp: "Formatted at 2026-06-20 20:19:10 on dist-test-slave-2hpg"
server_key: "322fa0b1cbcc51e6e4d913d5dde29c77"
server_key_iv: "12d9c585b627ebd71485d82d57331537"
server_key_version: "encryptionkey@0"
I20260620 20:19:10.058940 31338 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986536687152-17252-0/minicluster-data/ts-1/wal/instance:
uuid: "89eac4104e6c4573b766d41f6999395e"
format_stamp: "Formatted at 2026-06-20 20:19:10 on dist-test-slave-2hpg"
server_key: "322fa0b1cbcc51e6e4d913d5dde29c77"
server_key_iv: "12d9c585b627ebd71485d82d57331537"
server_key_version: "encryptionkey@0"
I20260620 20:19:10.064203 31338 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 20:19:10.067468 31353 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:19:10.068737 31338 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260620 20:19:10.068866 31338 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986536687152-17252-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986536687152-17252-0/minicluster-data/ts-1/wal
uuid: "89eac4104e6c4573b766d41f6999395e"
format_stamp: "Formatted at 2026-06-20 20:19:10 on dist-test-slave-2hpg"
server_key: "322fa0b1cbcc51e6e4d913d5dde29c77"
server_key_iv: "12d9c585b627ebd71485d82d57331537"
server_key_version: "encryptionkey@0"
I20260620 20:19:10.069039 31338 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986536687152-17252-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986536687152-17252-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986536687152-17252-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:19:10.091246 31338 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:19:10.095310 31338 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:19:10.095541 31338 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:19:10.096395 31338 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:19:10.097609 31338 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:19:10.097683 31338 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:19:10.097764 31338 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:19:10.097807 31338 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:19:10.130774 31338 rpc_server.cc:307] RPC server started. Bound to: 127.16.217.2:33683
I20260620 20:19:10.130867 31465 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.217.2:33683 every 8 connection(s)
I20260620 20:19:10.132025 31338 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986536687152-17252-0/minicluster-data/ts-1/data/info.pb
I20260620 20:19:10.139607 17252 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu as pid 31338
I20260620 20:19:10.139833 17252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986536687152-17252-0/minicluster-data/ts-1/wal/instance
I20260620 20:19:10.140419 17252 external_mini_cluster.cc:1428] Setting key 18058a9be1e67bcccef339fff7c8b65d
Jun 20 20:19:10 dist-test-slave-2hpg krb5kdc[31098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986750, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.217.2@KRBTEST.COM for kudu/127.16.217.62@KRBTEST.COM
I20260620 20:19:10.152632 31334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:19:10.134847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.217.2:33969 (local address 127.16.217.62:36477)
0620 20:19:10.135049 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0620 20:19:10.135054 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:19:10.136199 (+ 1145us) server_negotiation.cc:407] Connection header received
0620 20:19:10.138031 (+ 1832us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:19:10.138036 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:19:10.138118 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:19:10.138213 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:19:10.139812 (+ 1599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:19:10.140634 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:19:10.142257 (+ 1623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:19:10.142690 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:19:10.142832 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:19:10.146588 (+ 3756us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:19:10.146593 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:19:10.146596 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:19:10.146626 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:19:10.149378 (+ 2752us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:19:10.150255 (+ 877us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:19:10.150264 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:19:10.150267 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:19:10.150437 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:19:10.150862 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:19:10.150867 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:19:10.150871 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:19:10.151207 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:19:10.151359 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0620 20:19:10.152045 (+ 686us) server_negotiation.cc:299] Negotiation successful
0620 20:19:10.152290 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 20:19:10.153601 31468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:19:10.135341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:36477 (local address 127.16.217.2:33969)
0620 20:19:10.136034 (+ 693us) negotiation.cc:107] Waiting for socket to connect
0620 20:19:10.136071 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 20:19:10.137719 (+ 1648us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:19:10.138429 (+ 710us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:19:10.138438 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:19:10.138983 (+ 545us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:19:10.139598 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:19:10.139621 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:19:10.140795 (+ 1174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:19:10.140806 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:19:10.142099 (+ 1293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:19:10.142111 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:19:10.142913 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:19:10.142919 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:19:10.143147 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:19:10.143840 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:19:10.143863 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:19:10.146362 (+ 2499us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:19:10.149514 (+ 3152us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:19:10.149524 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:19:10.149541 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:19:10.150094 (+ 553us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:19:10.150545 (+ 451us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:19:10.150551 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:19:10.150555 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:19:10.150742 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:19:10.151328 (+ 586us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:19:10.151336 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:19:10.151674 (+ 338us) client_negotiation.cc:769] Sending connection context
0620 20:19:10.151933 (+ 259us) client_negotiation.cc:240] Negotiation successful
0620 20:19:10.152488 (+ 555us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":470,"thread_start_us":276,"threads_started":1}
I20260620 20:19:10.155161 31466 heartbeater.cc:347] Connected to a master server at 127.16.217.62:36477
I20260620 20:19:10.155525 31466 heartbeater.cc:464] Registering TS with master...
I20260620 20:19:10.156464 31466 heartbeater.cc:511] Master 127.16.217.62:36477 requested a full tablet report, sending...
I20260620 20:19:10.158258 31143 ts_manager.cc:194] Registered new tserver with Master: 89eac4104e6c4573b766d41f6999395e (127.16.217.2:33683)
I20260620 20:19:10.159731 31143 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.217.2@KRBTEST.COM'} at 127.16.217.2:33969
WARNING: no policy specified for kudu/127.16.217.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.217.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.217.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.3.keytab.
Entry for principal kudu/127.16.217.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.3.keytab.
I20260620 20:19:10.194478 17252 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
/tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986536687152-17252-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986536687152-17252-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986536687152-17252-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986536687152-17252-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.217.3:0
--local_ip_for_outbound_sockets=127.16.217.3
--webserver_interface=127.16.217.3
--webserver_port=0
--tserver_master_addrs=127.16.217.62:36477
--keytab_file=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.3.keytab
--principal=kudu/127.16.217.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.217.20:38487
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986536687152-17252-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986536687152-17252-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986536687152-17252-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:19:10.518970 31471 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:19:10.519249 31471 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:19:10.519414 31471 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:19:10.526980 31471 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:19:10.527086 31471 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:19:10.527264 31471 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.217.3
I20260620 20:19:10.537328 31471 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.217.20:38487
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986536687152-17252-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986536687152-17252-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.3.keytab
--principal=kudu/127.16.217.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.217.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986536687152-17252-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.217.3
--webserver_port=0
--tserver_master_addrs=127.16.217.62:36477
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.217.3
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986536687152-17252-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:19:10.538688 31471 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:19:10.540056 31471 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:19:10.554386 31476 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:19:10.554515 31477 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:19:10.554714 31479 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:19:10.555655 31471 server_base.cc:1034] running on GCE node
I20260620 20:19:10.556291 31471 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:19:10.557245 31471 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:19:10.558465 31471 hybrid_clock.cc:648] HybridClock initialized: now 1781986750558413 us; error 48 us; skew 500 ppm
I20260620 20:19:10.558826 31471 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:19:10 dist-test-slave-2hpg krb5kdc[31098](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986750, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.217.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:19:10.562217 31471 init.cc:377] Logged in from keytab as kudu/127.16.217.3@KRBTEST.COM (short username kudu)
I20260620 20:19:10.563941 31471 webserver.cc:460] Webserver started at http://127.16.217.3:39173/ using document root <none> and password file <none>
I20260620 20:19:10.564584 31471 fs_manager.cc:359] Metadata directory not provided
I20260620 20:19:10.564682 31471 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:19:10.564920 31471 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:19:10.567740 31471 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986536687152-17252-0/minicluster-data/ts-2/data/instance:
uuid: "5ee69ada9a214bc8a73ad8e534401c60"
format_stamp: "Formatted at 2026-06-20 20:19:10 on dist-test-slave-2hpg"
server_key: "affbce870a7beb9fcad9c5c2b6384c61"
server_key_iv: "9b939c1a3d3d41a24e1720b1879e1358"
server_key_version: "encryptionkey@0"
I20260620 20:19:10.568531 31471 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986536687152-17252-0/minicluster-data/ts-2/wal/instance:
uuid: "5ee69ada9a214bc8a73ad8e534401c60"
format_stamp: "Formatted at 2026-06-20 20:19:10 on dist-test-slave-2hpg"
server_key: "affbce870a7beb9fcad9c5c2b6384c61"
server_key_iv: "9b939c1a3d3d41a24e1720b1879e1358"
server_key_version: "encryptionkey@0"
I20260620 20:19:10.573644 31471 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260620 20:19:10.577105 31486 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:19:10.578593 31471 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 20:19:10.578791 31471 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986536687152-17252-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986536687152-17252-0/minicluster-data/ts-2/wal
uuid: "5ee69ada9a214bc8a73ad8e534401c60"
format_stamp: "Formatted at 2026-06-20 20:19:10 on dist-test-slave-2hpg"
server_key: "affbce870a7beb9fcad9c5c2b6384c61"
server_key_iv: "9b939c1a3d3d41a24e1720b1879e1358"
server_key_version: "encryptionkey@0"
I20260620 20:19:10.578987 31471 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986536687152-17252-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986536687152-17252-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986536687152-17252-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:19:10.591852 31471 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:19:10.596109 31471 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:19:10.596406 31471 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:19:10.597348 31471 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:19:10.598745 31471 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:19:10.598851 31471 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:19:10.599004 31471 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:19:10.599082 31471 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:19:10.633375 31471 rpc_server.cc:307] RPC server started. Bound to: 127.16.217.3:34191
I20260620 20:19:10.633536 31598 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.217.3:34191 every 8 connection(s)
I20260620 20:19:10.634897 31471 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986536687152-17252-0/minicluster-data/ts-2/data/info.pb
I20260620 20:19:10.637176 17252 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu as pid 31471
I20260620 20:19:10.637384 17252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986536687152-17252-0/minicluster-data/ts-2/wal/instance
I20260620 20:19:10.637930 17252 external_mini_cluster.cc:1428] Setting key 85d1e4ad2051c1b5e0f3efe89c12664b
I20260620 20:19:10.648003 31332 heartbeater.cc:503] Master 127.16.217.62:36477 was elected leader, sending a full tablet report...
Jun 20 20:19:10 dist-test-slave-2hpg krb5kdc[31098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986750, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.217.3@KRBTEST.COM for kudu/127.16.217.62@KRBTEST.COM
I20260620 20:19:10.655241 31334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:19:10.637630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.217.3:34577 (local address 127.16.217.62:36477)
0620 20:19:10.637846 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0620 20:19:10.637852 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:19:10.639256 (+ 1404us) server_negotiation.cc:407] Connection header received
0620 20:19:10.641545 (+ 2289us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:19:10.641550 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:19:10.641634 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:19:10.641753 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:19:10.643181 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:19:10.643930 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:19:10.645426 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:19:10.645819 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:19:10.645995 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:19:10.649263 (+ 3268us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:19:10.649268 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:19:10.649272 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:19:10.649302 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:19:10.652097 (+ 2795us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:19:10.652982 (+ 885us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:19:10.652988 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:19:10.652992 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:19:10.653176 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:19:10.653621 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:19:10.653625 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:19:10.653627 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:19:10.653941 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:19:10.654061 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0620 20:19:10.654777 (+ 716us) server_negotiation.cc:299] Negotiation successful
0620 20:19:10.654932 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260620 20:19:10.656164 31601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:19:10.638230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:36477 (local address 127.16.217.3:34577)
0620 20:19:10.639096 (+ 866us) negotiation.cc:107] Waiting for socket to connect
0620 20:19:10.639139 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 20:19:10.641277 (+ 2138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:19:10.642001 (+ 724us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:19:10.642009 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:19:10.642482 (+ 473us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:19:10.643019 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:19:10.643035 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:19:10.644073 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:19:10.644078 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:19:10.645275 (+ 1197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:19:10.645287 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:19:10.646072 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:19:10.646081 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:19:10.646328 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:19:10.646892 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:19:10.646912 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:19:10.649048 (+ 2136us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:19:10.652256 (+ 3208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:19:10.652267 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:19:10.652283 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:19:10.652872 (+ 589us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:19:10.653309 (+ 437us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:19:10.653313 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:19:10.653316 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:19:10.653519 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:19:10.654056 (+ 537us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:19:10.654062 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:19:10.654353 (+ 291us) client_negotiation.cc:769] Sending connection context
0620 20:19:10.654695 (+ 342us) client_negotiation.cc:240] Negotiation successful
0620 20:19:10.655083 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":570,"thread_start_us":342,"threads_started":1}
I20260620 20:19:10.657496 31599 heartbeater.cc:347] Connected to a master server at 127.16.217.62:36477
I20260620 20:19:10.657832 31599 heartbeater.cc:464] Registering TS with master...
I20260620 20:19:10.658582 31599 heartbeater.cc:511] Master 127.16.217.62:36477 requested a full tablet report, sending...
I20260620 20:19:10.660219 31143 ts_manager.cc:194] Registered new tserver with Master: 5ee69ada9a214bc8a73ad8e534401c60 (127.16.217.3:34191)
I20260620 20:19:10.661063 31143 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.217.3@KRBTEST.COM'} at 127.16.217.3:34577
I20260620 20:19:10.665740 17252 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 20:19:10.683300 31334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:19:10.670434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41010 (local address 127.16.217.62:36477)
0620 20:19:10.670658 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0620 20:19:10.670663 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:19:10.670993 (+ 330us) server_negotiation.cc:407] Connection header received
0620 20:19:10.671134 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:19:10.671138 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:19:10.671199 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:19:10.671286 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:19:10.672791 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:19:10.673600 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:19:10.675047 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:19:10.675461 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:19:10.675557 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:19:10.677796 (+ 2239us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:19:10.677800 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:19:10.677803 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:19:10.677832 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:19:10.680376 (+ 2544us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:19:10.681100 (+ 724us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:19:10.681104 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:19:10.681106 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:19:10.681241 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:19:10.681657 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:19:10.681662 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:19:10.681664 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:19:10.682181 (+ 517us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:19:10.682416 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0620 20:19:10.682751 (+ 335us) server_negotiation.cc:299] Negotiation successful
0620 20:19:10.682927 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 20:19:10.687633 31143 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41010:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 20:19:10.695010 31143 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 20:19:10.730870 31613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:19:10.707692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57822 (local address 127.16.217.1:44135)
0620 20:19:10.721141 (+ 13449us) server_negotiation.cc:206] Beginning negotiation
0620 20:19:10.721153 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 20:19:10.721189 (+ 36us) server_negotiation.cc:407] Connection header received
0620 20:19:10.721338 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:19:10.721349 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:19:10.721656 (+ 307us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:19:10.721901 (+ 245us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:19:10.723285 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:19:10.724936 (+ 1651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:19:10.728054 (+ 3118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:19:10.729537 (+ 1483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:19:10.729703 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:19:10.729987 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0620 20:19:10.730342 (+ 355us) server_negotiation.cc:299] Negotiation successful
0620 20:19:10.730520 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13226,"thread_start_us":14025,"threads_started":1}
I20260620 20:19:10.731076 31612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:19:10.707428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.1:44135 (local address 127.0.0.1:57822)
0620 20:19:10.718600 (+ 11172us) negotiation.cc:107] Waiting for socket to connect
0620 20:19:10.718666 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0620 20:19:10.718903 (+ 237us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:19:10.722399 (+ 3496us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:19:10.722405 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:19:10.722456 (+ 51us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:19:10.722974 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:19:10.722985 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:19:10.725074 (+ 2089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:19:10.725079 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:19:10.727861 (+ 2782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:19:10.727876 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:19:10.729800 (+ 1924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:19:10.729809 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:19:10.730041 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:19:10.730113 (+ 72us) client_negotiation.cc:769] Sending connection context
0620 20:19:10.730258 (+ 145us) client_negotiation.cc:240] Negotiation successful
0620 20:19:10.730474 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10807,"mutex_wait_us":133,"spinlock_wait_cycles":3328,"thread_start_us":14555,"threads_started":1}
I20260620 20:19:10.731386 31615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:19:10.708918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34602 (local address 127.16.217.3:34191)
0620 20:19:10.719897 (+ 10979us) server_negotiation.cc:206] Beginning negotiation
0620 20:19:10.719909 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 20:19:10.720729 (+ 820us) server_negotiation.cc:407] Connection header received
0620 20:19:10.720858 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:19:10.720866 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:19:10.721161 (+ 295us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:19:10.721413 (+ 252us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:19:10.723135 (+ 1722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:19:10.724919 (+ 1784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:19:10.728456 (+ 3537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:19:10.729710 (+ 1254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:19:10.729857 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:19:10.730131 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0620 20:19:10.730808 (+ 677us) server_negotiation.cc:299] Negotiation successful
0620 20:19:10.730987 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10769,"thread_start_us":10605,"threads_started":1}
I20260620 20:19:10.732108 31616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:19:10.709588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54014 (local address 127.16.217.2:33683)
0620 20:19:10.719332 (+ 9744us) server_negotiation.cc:206] Beginning negotiation
0620 20:19:10.719346 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 20:19:10.722443 (+ 3097us) server_negotiation.cc:407] Connection header received
0620 20:19:10.722651 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:19:10.722660 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:19:10.722892 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:19:10.723074 (+ 182us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:19:10.723741 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:19:10.725086 (+ 1345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:19:10.728423 (+ 3337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:19:10.729808 (+ 1385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:19:10.729978 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:19:10.730149 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0620 20:19:10.731620 (+ 1471us) server_negotiation.cc:299] Negotiation successful
0620 20:19:10.731780 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9570,"thread_start_us":9458,"threads_started":1}
I20260620 20:19:10.732179 31614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:19:10.707519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.3:34191 (local address 127.0.0.1:34602)
0620 20:19:10.720399 (+ 12880us) negotiation.cc:107] Waiting for socket to connect
0620 20:19:10.720419 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 20:19:10.720573 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:19:10.721535 (+ 962us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:19:10.721556 (+ 21us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:19:10.721590 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:19:10.722896 (+ 1306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:19:10.722912 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:19:10.725135 (+ 2223us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:19:10.725139 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:19:10.728249 (+ 3110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:19:10.728266 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:19:10.730351 (+ 2085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:19:10.730358 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:19:10.730530 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:19:10.730612 (+ 82us) client_negotiation.cc:769] Sending connection context
0620 20:19:10.731590 (+ 978us) client_negotiation.cc:240] Negotiation successful
0620 20:19:10.731740 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12744,"mutex_wait_us":194,"thread_start_us":10535,"threads_started":1}
I20260620 20:19:10.732601 31611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:19:10.708106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.2:33683 (local address 127.0.0.1:54014)
0620 20:19:10.722288 (+ 14182us) negotiation.cc:107] Waiting for socket to connect
0620 20:19:10.722307 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 20:19:10.722438 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:19:10.723112 (+ 674us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:19:10.723134 (+ 22us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:19:10.723162 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:19:10.723554 (+ 392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:19:10.723569 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:19:10.725248 (+ 1679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:19:10.725254 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:19:10.728268 (+ 3014us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:19:10.728278 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:19:10.730410 (+ 2132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:19:10.730419 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:19:10.730641 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:19:10.730679 (+ 38us) client_negotiation.cc:769] Sending connection context
0620 20:19:10.730767 (+ 88us) client_negotiation.cc:240] Negotiation successful
0620 20:19:10.730909 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14046,"mutex_wait_us":168,"thread_start_us":11999,"threads_started":1}
I20260620 20:19:10.735754 31267 tablet_service.cc:1467] Processing CreateTablet for tablet 58040bdc5ddc4da6b22b2195cdc0577e (DEFAULT_TABLE table=test-table [id=86f64c95c8934c02bd5154b96bf5a4ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:19:10.735857 31534 tablet_service.cc:1467] Processing CreateTablet for tablet 58040bdc5ddc4da6b22b2195cdc0577e (DEFAULT_TABLE table=test-table [id=86f64c95c8934c02bd5154b96bf5a4ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:19:10.736402 31401 tablet_service.cc:1467] Processing CreateTablet for tablet 58040bdc5ddc4da6b22b2195cdc0577e (DEFAULT_TABLE table=test-table [id=86f64c95c8934c02bd5154b96bf5a4ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:19:10.737181 31267 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 58040bdc5ddc4da6b22b2195cdc0577e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:19:10.737224 31534 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 58040bdc5ddc4da6b22b2195cdc0577e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:19:10.737593 31401 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 58040bdc5ddc4da6b22b2195cdc0577e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:19:10.749784 31617 tablet_bootstrap.cc:492] T 58040bdc5ddc4da6b22b2195cdc0577e P 8400677523bb4227b8e2d7b83e7f45ba: Bootstrap starting.
I20260620 20:19:10.752002 31618 tablet_bootstrap.cc:492] T 58040bdc5ddc4da6b22b2195cdc0577e P 5ee69ada9a214bc8a73ad8e534401c60: Bootstrap starting.
I20260620 20:19:10.753909 31617 tablet_bootstrap.cc:654] T 58040bdc5ddc4da6b22b2195cdc0577e P 8400677523bb4227b8e2d7b83e7f45ba: Neither blocks nor log segments found. Creating new log.
I20260620 20:19:10.755300 31617 log.cc:824] T 58040bdc5ddc4da6b22b2195cdc0577e P 8400677523bb4227b8e2d7b83e7f45ba: Log is configured to *not* fsync() on all Append() calls
I20260620 20:19:10.755940 31618 tablet_bootstrap.cc:654] T 58040bdc5ddc4da6b22b2195cdc0577e P 5ee69ada9a214bc8a73ad8e534401c60: Neither blocks nor log segments found. Creating new log.
I20260620 20:19:10.757606 31618 log.cc:824] T 58040bdc5ddc4da6b22b2195cdc0577e P 5ee69ada9a214bc8a73ad8e534401c60: Log is configured to *not* fsync() on all Append() calls
I20260620 20:19:10.758579 31619 tablet_bootstrap.cc:492] T 58040bdc5ddc4da6b22b2195cdc0577e P 89eac4104e6c4573b766d41f6999395e: Bootstrap starting.
I20260620 20:19:10.761262 31619 tablet_bootstrap.cc:654] T 58040bdc5ddc4da6b22b2195cdc0577e P 89eac4104e6c4573b766d41f6999395e: Neither blocks nor log segments found. Creating new log.
I20260620 20:19:10.763023 31619 log.cc:824] T 58040bdc5ddc4da6b22b2195cdc0577e P 89eac4104e6c4573b766d41f6999395e: Log is configured to *not* fsync() on all Append() calls
I20260620 20:19:10.764734 31617 tablet_bootstrap.cc:492] T 58040bdc5ddc4da6b22b2195cdc0577e P 8400677523bb4227b8e2d7b83e7f45ba: No bootstrap required, opened a new log
I20260620 20:19:10.765110 31617 ts_tablet_manager.cc:1397] T 58040bdc5ddc4da6b22b2195cdc0577e P 8400677523bb4227b8e2d7b83e7f45ba: Time spent bootstrapping tablet: real 0.016s user 0.003s sys 0.011s
I20260620 20:19:10.766021 31619 tablet_bootstrap.cc:492] T 58040bdc5ddc4da6b22b2195cdc0577e P 89eac4104e6c4573b766d41f6999395e: No bootstrap required, opened a new log
I20260620 20:19:10.766067 31618 tablet_bootstrap.cc:492] T 58040bdc5ddc4da6b22b2195cdc0577e P 5ee69ada9a214bc8a73ad8e534401c60: No bootstrap required, opened a new log
I20260620 20:19:10.766291 31619 ts_tablet_manager.cc:1397] T 58040bdc5ddc4da6b22b2195cdc0577e P 89eac4104e6c4573b766d41f6999395e: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260620 20:19:10.766372 31618 ts_tablet_manager.cc:1397] T 58040bdc5ddc4da6b22b2195cdc0577e P 5ee69ada9a214bc8a73ad8e534401c60: Time spent bootstrapping tablet: real 0.015s user 0.005s sys 0.007s
I20260620 20:19:10.773991 31619 raft_consensus.cc:348] T 58040bdc5ddc4da6b22b2195cdc0577e P 89eac4104e6c4573b766d41f6999395e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8400677523bb4227b8e2d7b83e7f45ba" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 44135 } } peers { permanent_uuid: "5ee69ada9a214bc8a73ad8e534401c60" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 34191 } } peers { permanent_uuid: "89eac4104e6c4573b766d41f6999395e" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 33683 } }
I20260620 20:19:10.774362 31619 raft_consensus.cc:374] T 58040bdc5ddc4da6b22b2195cdc0577e P 89eac4104e6c4573b766d41f6999395e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:19:10.774469 31619 raft_consensus.cc:729] T 58040bdc5ddc4da6b22b2195cdc0577e P 89eac4104e6c4573b766d41f6999395e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89eac4104e6c4573b766d41f6999395e, State: Initialized, Role: FOLLOWER
I20260620 20:19:10.775184 31619 consensus_queue.cc:260] T 58040bdc5ddc4da6b22b2195cdc0577e P 89eac4104e6c4573b766d41f6999395e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8400677523bb4227b8e2d7b83e7f45ba" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 44135 } } peers { permanent_uuid: "5ee69ada9a214bc8a73ad8e534401c60" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 34191 } } peers { permanent_uuid: "89eac4104e6c4573b766d41f6999395e" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 33683 } }
I20260620 20:19:10.776530 31617 raft_consensus.cc:348] T 58040bdc5ddc4da6b22b2195cdc0577e P 8400677523bb4227b8e2d7b83e7f45ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8400677523bb4227b8e2d7b83e7f45ba" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 44135 } } peers { permanent_uuid: "5ee69ada9a214bc8a73ad8e534401c60" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 34191 } } peers { permanent_uuid: "89eac4104e6c4573b766d41f6999395e" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 33683 } }
I20260620 20:19:10.777001 31617 raft_consensus.cc:374] T 58040bdc5ddc4da6b22b2195cdc0577e P 8400677523bb4227b8e2d7b83e7f45ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:19:10.777184 31617 raft_consensus.cc:729] T 58040bdc5ddc4da6b22b2195cdc0577e P 8400677523bb4227b8e2d7b83e7f45ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8400677523bb4227b8e2d7b83e7f45ba, State: Initialized, Role: FOLLOWER
I20260620 20:19:10.777680 31466 heartbeater.cc:503] Master 127.16.217.62:36477 was elected leader, sending a full tablet report...
I20260620 20:19:10.777981 31617 consensus_queue.cc:260] T 58040bdc5ddc4da6b22b2195cdc0577e P 8400677523bb4227b8e2d7b83e7f45ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8400677523bb4227b8e2d7b83e7f45ba" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 44135 } } peers { permanent_uuid: "5ee69ada9a214bc8a73ad8e534401c60" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 34191 } } peers { permanent_uuid: "89eac4104e6c4573b766d41f6999395e" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 33683 } }
I20260620 20:19:10.778018 31618 raft_consensus.cc:348] T 58040bdc5ddc4da6b22b2195cdc0577e P 5ee69ada9a214bc8a73ad8e534401c60 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8400677523bb4227b8e2d7b83e7f45ba" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 44135 } } peers { permanent_uuid: "5ee69ada9a214bc8a73ad8e534401c60" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 34191 } } peers { permanent_uuid: "89eac4104e6c4573b766d41f6999395e" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 33683 } }
I20260620 20:19:10.778466 31618 raft_consensus.cc:374] T 58040bdc5ddc4da6b22b2195cdc0577e P 5ee69ada9a214bc8a73ad8e534401c60 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:19:10.778666 31618 raft_consensus.cc:729] T 58040bdc5ddc4da6b22b2195cdc0577e P 5ee69ada9a214bc8a73ad8e534401c60 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ee69ada9a214bc8a73ad8e534401c60, State: Initialized, Role: FOLLOWER
I20260620 20:19:10.779388 31618 consensus_queue.cc:260] T 58040bdc5ddc4da6b22b2195cdc0577e P 5ee69ada9a214bc8a73ad8e534401c60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8400677523bb4227b8e2d7b83e7f45ba" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 44135 } } peers { permanent_uuid: "5ee69ada9a214bc8a73ad8e534401c60" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 34191 } } peers { permanent_uuid: "89eac4104e6c4573b766d41f6999395e" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 33683 } }
I20260620 20:19:10.779929 31617 ts_tablet_manager.cc:1428] T 58040bdc5ddc4da6b22b2195cdc0577e P 8400677523bb4227b8e2d7b83e7f45ba: Time spent starting tablet: real 0.015s user 0.014s sys 0.000s
I20260620 20:19:10.780975 31599 heartbeater.cc:503] Master 127.16.217.62:36477 was elected leader, sending a full tablet report...
I20260620 20:19:10.781651 31618 ts_tablet_manager.cc:1428] T 58040bdc5ddc4da6b22b2195cdc0577e P 5ee69ada9a214bc8a73ad8e534401c60: Time spent starting tablet: real 0.015s user 0.013s sys 0.000s
I20260620 20:19:10.781791 31619 ts_tablet_manager.cc:1428] T 58040bdc5ddc4da6b22b2195cdc0577e P 89eac4104e6c4573b766d41f6999395e: Time spent starting tablet: real 0.015s user 0.011s sys 0.000s
W20260620 20:19:10.866986 31333 tablet.cc:2369] T 58040bdc5ddc4da6b22b2195cdc0577e P 8400677523bb4227b8e2d7b83e7f45ba: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 20:19:10.884050 31467 tablet.cc:2369] T 58040bdc5ddc4da6b22b2195cdc0577e P 89eac4104e6c4573b766d41f6999395e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 20:19:10.886492 31600 tablet.cc:2369] T 58040bdc5ddc4da6b22b2195cdc0577e P 5ee69ada9a214bc8a73ad8e534401c60: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 20:19:10.923475 31624 raft_consensus.cc:482] T 58040bdc5ddc4da6b22b2195cdc0577e P 8400677523bb4227b8e2d7b83e7f45ba [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 20:19:10.923828 31624 raft_consensus.cc:504] T 58040bdc5ddc4da6b22b2195cdc0577e P 8400677523bb4227b8e2d7b83e7f45ba [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8400677523bb4227b8e2d7b83e7f45ba" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 44135 } } peers { permanent_uuid: "5ee69ada9a214bc8a73ad8e534401c60" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 34191 } } peers { permanent_uuid: "89eac4104e6c4573b766d41f6999395e" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 33683 } }
I20260620 20:19:10.925503 31624 leader_election.cc:288] T 58040bdc5ddc4da6b22b2195cdc0577e P 8400677523bb4227b8e2d7b83e7f45ba [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5ee69ada9a214bc8a73ad8e534401c60 (127.16.217.3:34191), 89eac4104e6c4573b766d41f6999395e (127.16.217.2:33683)
I20260620 20:19:10.932170 31616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:19:10.925988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.217.1:40253 (local address 127.16.217.2:33683)
0620 20:19:10.926246 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0620 20:19:10.926251 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:19:10.926569 (+ 318us) server_negotiation.cc:407] Connection header received
0620 20:19:10.926806 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:19:10.926812 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:19:10.926882 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:19:10.926975 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:19:10.927532 (+ 557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:19:10.928242 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:19:10.930341 (+ 2099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:19:10.931096 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:19:10.931283 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:19:10.931361 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0620 20:19:10.931733 (+ 372us) server_negotiation.cc:299] Negotiation successful
0620 20:19:10.931860 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260620 20:19:10.932258 31615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:19:10.925978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.217.1:54087 (local address 127.16.217.3:34191)
0620 20:19:10.926222 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0620 20:19:10.926230 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 20:19:10.926997 (+ 767us) server_negotiation.cc:407] Connection header received
0620 20:19:10.927217 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:19:10.927224 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:19:10.927286 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:19:10.927384 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:19:10.927935 (+ 551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:19:10.928660 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:19:10.930491 (+ 1831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:19:10.931231 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:19:10.931418 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:19:10.931495 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 20:19:10.931880 (+ 385us) server_negotiation.cc:299] Negotiation successful
0620 20:19:10.932000 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260620 20:19:10.932327 31626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:19:10.925797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.2:33683 (local address 127.16.217.1:40253)
0620 20:19:10.926452 (+ 655us) negotiation.cc:107] Waiting for socket to connect
0620 20:19:10.926472 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 20:19:10.926656 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:19:10.927004 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:19:10.927009 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:19:10.927094 (+ 85us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:19:10.927389 (+ 295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:19:10.927398 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:19:10.928368 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:19:10.928376 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:19:10.930184 (+ 1808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:19:10.930196 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:19:10.931307 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:19:10.931313 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:19:10.931547 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:19:10.931582 (+ 35us) client_negotiation.cc:769] Sending connection context
0620 20:19:10.931695 (+ 113us) client_negotiation.cc:240] Negotiation successful
0620 20:19:10.931823 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":538,"mutex_wait_us":22,"spinlock_wait_cycles":23296,"thread_start_us":328,"threads_started":1}
I20260620 20:19:10.932438 31627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:19:10.925797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.3:34191 (local address 127.16.217.1:54087)
0620 20:19:10.926878 (+ 1081us) negotiation.cc:107] Waiting for socket to connect
0620 20:19:10.926892 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 20:19:10.927044 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:19:10.927414 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:19:10.927418 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:19:10.927441 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:19:10.927770 (+ 329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:19:10.927778 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:19:10.928798 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:19:10.928804 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:19:10.930337 (+ 1533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:19:10.930349 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:19:10.931448 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:19:10.931454 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:19:10.931708 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:19:10.931730 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 20:19:10.931838 (+ 108us) client_negotiation.cc:240] Negotiation successful
0620 20:19:10.931962 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":983,"mutex_wait_us":538,"thread_start_us":315,"threads_started":1}
I20260620 20:19:10.933357 31554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58040bdc5ddc4da6b22b2195cdc0577e" candidate_uuid: "8400677523bb4227b8e2d7b83e7f45ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ee69ada9a214bc8a73ad8e534401c60" is_pre_election: true
I20260620 20:19:10.933341 31421 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58040bdc5ddc4da6b22b2195cdc0577e" candidate_uuid: "8400677523bb4227b8e2d7b83e7f45ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89eac4104e6c4573b766d41f6999395e" is_pre_election: true
I20260620 20:19:10.933840 31421 raft_consensus.cc:2454] T 58040bdc5ddc4da6b22b2195cdc0577e P 89eac4104e6c4573b766d41f6999395e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8400677523bb4227b8e2d7b83e7f45ba in term 0.
I20260620 20:19:10.933843 31554 raft_consensus.cc:2454] T 58040bdc5ddc4da6b22b2195cdc0577e P 5ee69ada9a214bc8a73ad8e534401c60 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8400677523bb4227b8e2d7b83e7f45ba in term 0.
I20260620 20:19:10.934573 31220 leader_election.cc:302] T 58040bdc5ddc4da6b22b2195cdc0577e P 8400677523bb4227b8e2d7b83e7f45ba [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 5ee69ada9a214bc8a73ad8e534401c60, 8400677523bb4227b8e2d7b83e7f45ba, 89eac4104e6c4573b766d41f6999395e; no voters:
I20260620 20:19:10.934966 31624 raft_consensus.cc:2781] T 58040bdc5ddc4da6b22b2195cdc0577e P 8400677523bb4227b8e2d7b83e7f45ba [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 20:19:10.935115 31624 raft_consensus.cc:482] T 58040bdc5ddc4da6b22b2195cdc0577e P 8400677523bb4227b8e2d7b83e7f45ba [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 20:19:10.935230 31624 raft_consensus.cc:3037] T 58040bdc5ddc4da6b22b2195cdc0577e P 8400677523bb4227b8e2d7b83e7f45ba [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:19:10.937279 31624 raft_consensus.cc:504] T 58040bdc5ddc4da6b22b2195cdc0577e P 8400677523bb4227b8e2d7b83e7f45ba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8400677523bb4227b8e2d7b83e7f45ba" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 44135 } } peers { permanent_uuid: "5ee69ada9a214bc8a73ad8e534401c60" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 34191 } } peers { permanent_uuid: "89eac4104e6c4573b766d41f6999395e" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 33683 } }
I20260620 20:19:10.937822 31624 leader_election.cc:288] T 58040bdc5ddc4da6b22b2195cdc0577e P 8400677523bb4227b8e2d7b83e7f45ba [CANDIDATE]: Term 1 election: Requested vote from peers 5ee69ada9a214bc8a73ad8e534401c60 (127.16.217.3:34191), 89eac4104e6c4573b766d41f6999395e (127.16.217.2:33683)
I20260620 20:19:10.938228 31554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58040bdc5ddc4da6b22b2195cdc0577e" candidate_uuid: "8400677523bb4227b8e2d7b83e7f45ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ee69ada9a214bc8a73ad8e534401c60"
I20260620 20:19:10.938269 31421 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58040bdc5ddc4da6b22b2195cdc0577e" candidate_uuid: "8400677523bb4227b8e2d7b83e7f45ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89eac4104e6c4573b766d41f6999395e"
I20260620 20:19:10.938416 31554 raft_consensus.cc:3037] T 58040bdc5ddc4da6b22b2195cdc0577e P 5ee69ada9a214bc8a73ad8e534401c60 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:19:10.938447 31421 raft_consensus.cc:3037] T 58040bdc5ddc4da6b22b2195cdc0577e P 89eac4104e6c4573b766d41f6999395e [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:19:10.940578 31421 raft_consensus.cc:2454] T 58040bdc5ddc4da6b22b2195cdc0577e P 89eac4104e6c4573b766d41f6999395e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8400677523bb4227b8e2d7b83e7f45ba in term 1.
I20260620 20:19:10.940963 31220 leader_election.cc:302] T 58040bdc5ddc4da6b22b2195cdc0577e P 8400677523bb4227b8e2d7b83e7f45ba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8400677523bb4227b8e2d7b83e7f45ba, 89eac4104e6c4573b766d41f6999395e; no voters:
I20260620 20:19:10.941013 31554 raft_consensus.cc:2454] T 58040bdc5ddc4da6b22b2195cdc0577e P 5ee69ada9a214bc8a73ad8e534401c60 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8400677523bb4227b8e2d7b83e7f45ba in term 1.
I20260620 20:19:10.941212 31624 raft_consensus.cc:2781] T 58040bdc5ddc4da6b22b2195cdc0577e P 8400677523bb4227b8e2d7b83e7f45ba [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:19:10.941663 31624 raft_consensus.cc:686] T 58040bdc5ddc4da6b22b2195cdc0577e P 8400677523bb4227b8e2d7b83e7f45ba [term 1 LEADER]: Becoming Leader. State: Replica: 8400677523bb4227b8e2d7b83e7f45ba, State: Running, Role: LEADER
I20260620 20:19:10.942219 31624 consensus_queue.cc:237] T 58040bdc5ddc4da6b22b2195cdc0577e P 8400677523bb4227b8e2d7b83e7f45ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8400677523bb4227b8e2d7b83e7f45ba" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 44135 } } peers { permanent_uuid: "5ee69ada9a214bc8a73ad8e534401c60" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 34191 } } peers { permanent_uuid: "89eac4104e6c4573b766d41f6999395e" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 33683 } }
I20260620 20:19:10.946552 31142 catalog_manager.cc:5495] T 58040bdc5ddc4da6b22b2195cdc0577e P 8400677523bb4227b8e2d7b83e7f45ba reported cstate change: term changed from 0 to 1, leader changed from <none> to 8400677523bb4227b8e2d7b83e7f45ba (127.16.217.1). New cstate: current_term: 1 leader_uuid: "8400677523bb4227b8e2d7b83e7f45ba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8400677523bb4227b8e2d7b83e7f45ba" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 44135 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5ee69ada9a214bc8a73ad8e534401c60" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 34191 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "89eac4104e6c4573b766d41f6999395e" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 33683 } health_report { overall_health: UNKNOWN } } }
I20260620 20:19:10.974541 31613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:19:10.969059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57834 (local address 127.16.217.1:44135)
0620 20:19:10.969291 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0620 20:19:10.969299 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 20:19:10.969318 (+ 19us) server_negotiation.cc:407] Connection header received
0620 20:19:10.969407 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:19:10.969411 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:19:10.969485 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:19:10.969646 (+ 161us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 20:19:10.970059 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:19:10.970725 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:19:10.972220 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:19:10.972774 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:19:10.972909 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:19:10.973402 (+ 493us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 20:19:10.973961 (+ 559us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 20:19:10.974109 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0620 20:19:10.974185 (+ 76us) server_negotiation.cc:299] Negotiation successful
0620 20:19:10.974266 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260620 20:19:10.982393 31554 raft_consensus.cc:1261] T 58040bdc5ddc4da6b22b2195cdc0577e P 5ee69ada9a214bc8a73ad8e534401c60 [term 1 FOLLOWER]: Refusing update from remote peer 8400677523bb4227b8e2d7b83e7f45ba: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:19:10.982393 31421 raft_consensus.cc:1261] T 58040bdc5ddc4da6b22b2195cdc0577e P 89eac4104e6c4573b766d41f6999395e [term 1 FOLLOWER]: Refusing update from remote peer 8400677523bb4227b8e2d7b83e7f45ba: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:19:10.983382 31628 consensus_queue.cc:1035] T 58040bdc5ddc4da6b22b2195cdc0577e P 8400677523bb4227b8e2d7b83e7f45ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ee69ada9a214bc8a73ad8e534401c60" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 34191 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:19:10.983708 31624 consensus_queue.cc:1035] T 58040bdc5ddc4da6b22b2195cdc0577e P 8400677523bb4227b8e2d7b83e7f45ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "89eac4104e6c4573b766d41f6999395e" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 33683 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:19:10.997023 31633 mvcc.cc:205] Tried to move back new op lower bound from 7299017732014067712 to 7299017731863044096. Current Snapshot: MvccSnapshot[applied={T|T < 7299017732014067712}]
I20260620 20:19:10.997110 31631 mvcc.cc:205] Tried to move back new op lower bound from 7299017732014067712 to 7299017731863044096. Current Snapshot: MvccSnapshot[applied={T|T < 7299017732014067712}]
I20260620 20:19:11.021570 31632 mvcc.cc:205] Tried to move back new op lower bound from 7299017732014067712 to 7299017731863044096. Current Snapshot: MvccSnapshot[applied={T|T < 7299017732014067712}]
I20260620 20:19:11.026909 31143 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41010:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 20:19:11.027246 31143 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41010:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 20:19:11.030776 31143 catalog_manager.cc:5782] T 00000000000000000000000000000000 P becbda298fcf43ab9c61810d35e9764a: Sending DeleteTablet for 3 replicas of tablet 58040bdc5ddc4da6b22b2195cdc0577e
I20260620 20:19:11.031903 31534 tablet_service.cc:1514] Processing DeleteTablet for tablet 58040bdc5ddc4da6b22b2195cdc0577e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:19:11 UTC) from {username='kudu', principal='kudu/127.16.217.62@KRBTEST.COM'} at 127.0.0.1:34602
I20260620 20:19:11.032017 31267 tablet_service.cc:1514] Processing DeleteTablet for tablet 58040bdc5ddc4da6b22b2195cdc0577e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:19:11 UTC) from {username='kudu', principal='kudu/127.16.217.62@KRBTEST.COM'} at 127.0.0.1:57822
I20260620 20:19:11.032855 31641 tablet_replica.cc:330] T 58040bdc5ddc4da6b22b2195cdc0577e P 5ee69ada9a214bc8a73ad8e534401c60: stopping tablet replica
I20260620 20:19:11.033020 31401 tablet_service.cc:1514] Processing DeleteTablet for tablet 58040bdc5ddc4da6b22b2195cdc0577e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:19:11 UTC) from {username='kudu', principal='kudu/127.16.217.62@KRBTEST.COM'} at 127.0.0.1:54014
I20260620 20:19:11.033136 17252 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu pid 31204
I20260620 20:19:11.033465 31641 raft_consensus.cc:2229] T 58040bdc5ddc4da6b22b2195cdc0577e P 5ee69ada9a214bc8a73ad8e534401c60 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 20:19:11.034171 31641 raft_consensus.cc:2258] T 58040bdc5ddc4da6b22b2195cdc0577e P 5ee69ada9a214bc8a73ad8e534401c60 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:19:11.036394 31641 ts_tablet_manager.cc:1905] T 58040bdc5ddc4da6b22b2195cdc0577e P 5ee69ada9a214bc8a73ad8e534401c60: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:19:11.045720 31641 ts_tablet_manager.cc:1918] T 58040bdc5ddc4da6b22b2195cdc0577e P 5ee69ada9a214bc8a73ad8e534401c60: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 20:19:11.046029 31641 log.cc:1196] T 58040bdc5ddc4da6b22b2195cdc0577e P 5ee69ada9a214bc8a73ad8e534401c60: Deleting WAL directory at /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986536687152-17252-0/minicluster-data/ts-2/wal/wals/58040bdc5ddc4da6b22b2195cdc0577e
I20260620 20:19:11.046772 31641 ts_tablet_manager.cc:1939] T 58040bdc5ddc4da6b22b2195cdc0577e P 5ee69ada9a214bc8a73ad8e534401c60: Deleting consensus metadata
I20260620 20:19:11.048625 31643 tablet_replica.cc:330] T 58040bdc5ddc4da6b22b2195cdc0577e P 89eac4104e6c4573b766d41f6999395e: stopping tablet replica
I20260620 20:19:11.048987 31130 catalog_manager.cc:4841] TS 5ee69ada9a214bc8a73ad8e534401c60 (127.16.217.3:34191): tablet 58040bdc5ddc4da6b22b2195cdc0577e (table test-table [id=86f64c95c8934c02bd5154b96bf5a4ef]) successfully deleted
I20260620 20:19:11.049069 31643 raft_consensus.cc:2229] T 58040bdc5ddc4da6b22b2195cdc0577e P 89eac4104e6c4573b766d41f6999395e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 20:19:11.049583 31643 raft_consensus.cc:2258] T 58040bdc5ddc4da6b22b2195cdc0577e P 89eac4104e6c4573b766d41f6999395e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:19:11.051553 31643 ts_tablet_manager.cc:1905] T 58040bdc5ddc4da6b22b2195cdc0577e P 89eac4104e6c4573b766d41f6999395e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:19:11.055262 31642 tablet_replica.cc:330] T 58040bdc5ddc4da6b22b2195cdc0577e P 8400677523bb4227b8e2d7b83e7f45ba: stopping tablet replica
I20260620 20:19:11.055748 31642 raft_consensus.cc:2229] T 58040bdc5ddc4da6b22b2195cdc0577e P 8400677523bb4227b8e2d7b83e7f45ba [term 1 LEADER]: Raft consensus shutting down.
I20260620 20:19:11.056475 31642 raft_consensus.cc:2258] T 58040bdc5ddc4da6b22b2195cdc0577e P 8400677523bb4227b8e2d7b83e7f45ba [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:19:11.057526 31643 ts_tablet_manager.cc:1918] T 58040bdc5ddc4da6b22b2195cdc0577e P 89eac4104e6c4573b766d41f6999395e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 20:19:11.057693 31643 log.cc:1196] T 58040bdc5ddc4da6b22b2195cdc0577e P 89eac4104e6c4573b766d41f6999395e: Deleting WAL directory at /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986536687152-17252-0/minicluster-data/ts-1/wal/wals/58040bdc5ddc4da6b22b2195cdc0577e
I20260620 20:19:11.058238 31643 ts_tablet_manager.cc:1939] T 58040bdc5ddc4da6b22b2195cdc0577e P 89eac4104e6c4573b766d41f6999395e: Deleting consensus metadata
I20260620 20:19:11.058625 31642 ts_tablet_manager.cc:1905] T 58040bdc5ddc4da6b22b2195cdc0577e P 8400677523bb4227b8e2d7b83e7f45ba: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:19:11.059973 31127 catalog_manager.cc:4841] TS 89eac4104e6c4573b766d41f6999395e (127.16.217.2:33683): tablet 58040bdc5ddc4da6b22b2195cdc0577e (table test-table [id=86f64c95c8934c02bd5154b96bf5a4ef]) successfully deleted
I20260620 20:19:11.064930 31642 ts_tablet_manager.cc:1918] T 58040bdc5ddc4da6b22b2195cdc0577e P 8400677523bb4227b8e2d7b83e7f45ba: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 20:19:11.065128 31642 log.cc:1196] T 58040bdc5ddc4da6b22b2195cdc0577e P 8400677523bb4227b8e2d7b83e7f45ba: Deleting WAL directory at /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986536687152-17252-0/minicluster-data/ts-0/wal/wals/58040bdc5ddc4da6b22b2195cdc0577e
I20260620 20:19:11.065594 31642 ts_tablet_manager.cc:1939] T 58040bdc5ddc4da6b22b2195cdc0577e P 8400677523bb4227b8e2d7b83e7f45ba: Deleting consensus metadata
Jun 20 20:19:11 dist-test-slave-2hpg krb5kdc[31098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986748, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.217.1@KRBTEST.COM
I20260620 20:19:11.067193 31129 catalog_manager.cc:4841] TS 8400677523bb4227b8e2d7b83e7f45ba (127.16.217.1:44135): tablet 58040bdc5ddc4da6b22b2195cdc0577e (table test-table [id=86f64c95c8934c02bd5154b96bf5a4ef]) successfully deleted
I20260620 20:19:11.074008 31613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:19:11.034344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57848 (local address 127.16.217.1:44135)
0620 20:19:11.034507 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0620 20:19:11.034513 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:19:11.054543 (+ 20030us) server_negotiation.cc:407] Connection header received
0620 20:19:11.054737 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:19:11.054744 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:19:11.054885 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:19:11.055013 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:19:11.057654 (+ 2641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:19:11.058872 (+ 1218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:19:11.060924 (+ 2052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:19:11.061679 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:19:11.061822 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:19:11.067575 (+ 5753us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:19:11.067586 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:19:11.067601 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:19:11.067637 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:19:11.070831 (+ 3194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:19:11.071668 (+ 837us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:19:11.071675 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:19:11.071677 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:19:11.071819 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:19:11.072318 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:19:11.072322 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:19:11.072324 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:19:11.072769 (+ 445us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:19:11.072982 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0620 20:19:11.073420 (+ 438us) server_negotiation.cc:299] Negotiation successful
0620 20:19:11.073661 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 20:19:11.074777 31327 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:19:11.175572 17252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu with pid 31204
I20260620 20:19:11.196931 17252 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu pid 31338
Jun 20 20:19:11 dist-test-slave-2hpg krb5kdc[31098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986748, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.217.2@KRBTEST.COM
I20260620 20:19:11.213054 31616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:19:11.198243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54024 (local address 127.16.217.2:33683)
0620 20:19:11.198412 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0620 20:19:11.198417 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:19:11.198433 (+ 16us) server_negotiation.cc:407] Connection header received
0620 20:19:11.198490 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:19:11.198494 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:19:11.198562 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:19:11.198651 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:19:11.200212 (+ 1561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:19:11.200953 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:19:11.202513 (+ 1560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:19:11.202902 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:19:11.202997 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:19:11.207171 (+ 4174us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:19:11.207186 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:19:11.207202 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:19:11.207236 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:19:11.210310 (+ 3074us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:19:11.211045 (+ 735us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:19:11.211053 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:19:11.211055 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:19:11.211181 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:19:11.211563 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:19:11.211567 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:19:11.211570 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:19:11.211958 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:19:11.212191 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0620 20:19:11.212484 (+ 293us) server_negotiation.cc:299] Negotiation successful
0620 20:19:11.212670 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 20:19:11.213835 31461 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:19:11.305290 17252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu with pid 31338
I20260620 20:19:11.322474 17252 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu pid 31471
Jun 20 20:19:11 dist-test-slave-2hpg krb5kdc[31098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986748, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.217.3@KRBTEST.COM
I20260620 20:19:11.338778 31615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:19:11.323313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34610 (local address 127.16.217.3:34191)
0620 20:19:11.323490 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0620 20:19:11.323495 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:19:11.323514 (+ 19us) server_negotiation.cc:407] Connection header received
0620 20:19:11.323663 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:19:11.323667 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:19:11.323731 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:19:11.323817 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:19:11.325111 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:19:11.325798 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:19:11.327148 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:19:11.327532 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:19:11.327624 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:19:11.332763 (+ 5139us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:19:11.332781 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:19:11.332802 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:19:11.332843 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:19:11.335877 (+ 3034us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:19:11.336613 (+ 736us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:19:11.336621 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:19:11.336623 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:19:11.336768 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:19:11.337152 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:19:11.337159 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:19:11.337162 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:19:11.337522 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:19:11.337828 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0620 20:19:11.338214 (+ 386us) server_negotiation.cc:299] Negotiation successful
0620 20:19:11.338429 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 20:19:11.339464 31594 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:19:11.433336 17252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu with pid 31471
I20260620 20:19:11.450376 17252 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu pid 31111
I20260620 20:19:11.451520 31173 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:19:11.565222 17252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu with pid 31111
I20260620 20:19:11.586619 17252 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
/tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986536687152-17252-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986536687152-17252-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986536687152-17252-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986536687152-17252-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.217.62:36477
--webserver_interface=127.16.217.62
--webserver_port=36691
--keytab_file=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.62.keytab
--principal=kudu/127.16.217.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.217.20:38487
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.217.62:36477
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986536687152-17252-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986536687152-17252-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986536687152-17252-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:19:11.911978 31649 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:19:11.912333 31649 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:19:11.912514 31649 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:19:11.919835 31649 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:19:11.919991 31649 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:19:11.920110 31649 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:19:11.920208 31649 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:19:11.920284 31649 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:19:11.930286 31649 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.217.20:38487
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986536687152-17252-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986536687152-17252-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.16.217.62:36477
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.62.keytab
--principal=kudu/127.16.217.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.217.62:36477
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986536687152-17252-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.217.62
--webserver_port=36691
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986536687152-17252-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:19:11.931612 31649 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:19:11.932989 31649 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:19:11.956276 31655 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:19:11.956362 31658 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:19:11.956314 31656 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:19:11.957036 31649 server_base.cc:1034] running on GCE node
I20260620 20:19:11.957993 31649 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:19:11.959580 31649 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:19:11.960836 31649 hybrid_clock.cc:648] HybridClock initialized: now 1781986751960788 us; error 26 us; skew 500 ppm
I20260620 20:19:11.961323 31649 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:19:11 dist-test-slave-2hpg krb5kdc[31098](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986751, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.217.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:19:11.965061 31649 init.cc:377] Logged in from keytab as kudu/127.16.217.62@KRBTEST.COM (short username kudu)
I20260620 20:19:11.966796 31649 webserver.cc:460] Webserver started at http://127.16.217.62:36691/ using document root <none> and password file <none>
I20260620 20:19:11.967694 31649 fs_manager.cc:359] Metadata directory not provided
I20260620 20:19:11.967934 31649 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:19:11.974215 31649 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.001s sys 0.003s
I20260620 20:19:11.977200 31665 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:19:11.979041 31649 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 20:19:11.979251 31649 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986536687152-17252-0/minicluster-data/master-0/data,/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986536687152-17252-0/minicluster-data/master-0/wal
uuid: "becbda298fcf43ab9c61810d35e9764a"
format_stamp: "Formatted at 2026-06-20 20:19:08 on dist-test-slave-2hpg"
server_key: "c6d40976d637a0ffe5a5a7c91c0ba6c9"
server_key_iv: "7208137db4a722e0cbcf6f8b467ef330"
server_key_version: "encryptionkey@0"
I20260620 20:19:11.980367 31649 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986536687152-17252-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986536687152-17252-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986536687152-17252-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:19:12.000804 31649 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:19:12.005111 31649 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:19:12.005420 31649 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:19:12.025209 31649 rpc_server.cc:307] RPC server started. Bound to: 127.16.217.62:36477
I20260620 20:19:12.025228 31716 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.217.62:36477 every 8 connection(s)
I20260620 20:19:12.026901 31649 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986536687152-17252-0/minicluster-data/master-0/data/info.pb
I20260620 20:19:12.029277 17252 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu as pid 31649
I20260620 20:19:12.030166 17252 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
/tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986536687152-17252-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986536687152-17252-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986536687152-17252-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986536687152-17252-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.217.1:44135
--local_ip_for_outbound_sockets=127.16.217.1
--tserver_master_addrs=127.16.217.62:36477
--webserver_port=38787
--webserver_interface=127.16.217.1
--keytab_file=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.1.keytab
--principal=kudu/127.16.217.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.217.20:38487
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986536687152-17252-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986536687152-17252-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986536687152-17252-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 20:19:12.034423 31717 sys_catalog.cc:263] Verifying existing consensus state
I20260620 20:19:12.037714 31717 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P becbda298fcf43ab9c61810d35e9764a: Bootstrap starting.
I20260620 20:19:12.057474 31717 log.cc:824] T 00000000000000000000000000000000 P becbda298fcf43ab9c61810d35e9764a: Log is configured to *not* fsync() on all Append() calls
I20260620 20:19:12.069691 31717 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P becbda298fcf43ab9c61810d35e9764a: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 20:19:12.070340 31717 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P becbda298fcf43ab9c61810d35e9764a: Bootstrap complete.
I20260620 20:19:12.079993 31717 raft_consensus.cc:348] T 00000000000000000000000000000000 P becbda298fcf43ab9c61810d35e9764a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "becbda298fcf43ab9c61810d35e9764a" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 36477 } }
I20260620 20:19:12.080888 31717 raft_consensus.cc:729] T 00000000000000000000000000000000 P becbda298fcf43ab9c61810d35e9764a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: becbda298fcf43ab9c61810d35e9764a, State: Initialized, Role: FOLLOWER
I20260620 20:19:12.081529 31717 consensus_queue.cc:260] T 00000000000000000000000000000000 P becbda298fcf43ab9c61810d35e9764a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "becbda298fcf43ab9c61810d35e9764a" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 36477 } }
I20260620 20:19:12.081785 31717 raft_consensus.cc:388] T 00000000000000000000000000000000 P becbda298fcf43ab9c61810d35e9764a [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 20:19:12.081938 31717 raft_consensus.cc:482] T 00000000000000000000000000000000 P becbda298fcf43ab9c61810d35e9764a [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 20:19:12.082123 31717 raft_consensus.cc:3037] T 00000000000000000000000000000000 P becbda298fcf43ab9c61810d35e9764a [term 1 FOLLOWER]: Advancing to term 2
I20260620 20:19:12.086174 31717 raft_consensus.cc:504] T 00000000000000000000000000000000 P becbda298fcf43ab9c61810d35e9764a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "becbda298fcf43ab9c61810d35e9764a" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 36477 } }
I20260620 20:19:12.086676 31717 leader_election.cc:302] T 00000000000000000000000000000000 P becbda298fcf43ab9c61810d35e9764a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: becbda298fcf43ab9c61810d35e9764a; no voters:
I20260620 20:19:12.087349 31717 leader_election.cc:288] T 00000000000000000000000000000000 P becbda298fcf43ab9c61810d35e9764a [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 20:19:12.087487 31721 raft_consensus.cc:2781] T 00000000000000000000000000000000 P becbda298fcf43ab9c61810d35e9764a [term 2 FOLLOWER]: Leader election won for term 2
I20260620 20:19:12.089954 31717 sys_catalog.cc:564] T 00000000000000000000000000000000 P becbda298fcf43ab9c61810d35e9764a [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:19:12.094182 31721 raft_consensus.cc:686] T 00000000000000000000000000000000 P becbda298fcf43ab9c61810d35e9764a [term 2 LEADER]: Becoming Leader. State: Replica: becbda298fcf43ab9c61810d35e9764a, State: Running, Role: LEADER
I20260620 20:19:12.094915 31721 consensus_queue.cc:237] T 00000000000000000000000000000000 P becbda298fcf43ab9c61810d35e9764a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "becbda298fcf43ab9c61810d35e9764a" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 36477 } }
I20260620 20:19:12.097707 31722 sys_catalog.cc:455] T 00000000000000000000000000000000 P becbda298fcf43ab9c61810d35e9764a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "becbda298fcf43ab9c61810d35e9764a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "becbda298fcf43ab9c61810d35e9764a" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 36477 } } }
I20260620 20:19:12.098184 31722 sys_catalog.cc:458] T 00000000000000000000000000000000 P becbda298fcf43ab9c61810d35e9764a [sys.catalog]: This master's current role is: LEADER
I20260620 20:19:12.098863 31723 sys_catalog.cc:455] T 00000000000000000000000000000000 P becbda298fcf43ab9c61810d35e9764a [sys.catalog]: SysCatalogTable state changed. Reason: New leader becbda298fcf43ab9c61810d35e9764a. Latest consensus state: current_term: 2 leader_uuid: "becbda298fcf43ab9c61810d35e9764a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "becbda298fcf43ab9c61810d35e9764a" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 36477 } } }
I20260620 20:19:12.099181 31723 sys_catalog.cc:458] T 00000000000000000000000000000000 P becbda298fcf43ab9c61810d35e9764a [sys.catalog]: This master's current role is: LEADER
I20260620 20:19:12.102798 31733 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 20:19:12.110800 31733 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 20:19:12.112577 31733 catalog_manager.cc:1259] Loaded cluster ID: a30997bc2e0d4a0b86613e6b28572b3a
I20260620 20:19:12.112704 31733 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 20:19:12.116741 31733 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260620 20:19:12.130899 31733 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260620 20:19:12.138023 31733 catalog_manager.cc:1504] Loading token signing keys...
I20260620 20:19:12.139297 31733 catalog_manager.cc:5879] T 00000000000000000000000000000000 P becbda298fcf43ab9c61810d35e9764a: Loaded TSK: 0
I20260620 20:19:12.140377 31733 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 20:19:12.401815 31719 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:19:12.402233 31719 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:19:12.402416 31719 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:19:12.409708 31719 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:19:12.409874 31719 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:19:12.410106 31719 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.217.1
I20260620 20:19:12.419992 31719 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.217.20:38487
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986536687152-17252-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986536687152-17252-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.1.keytab
--principal=kudu/127.16.217.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.217.1:44135
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986536687152-17252-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.217.1
--webserver_port=38787
--tserver_master_addrs=127.16.217.62:36477
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.217.1
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986536687152-17252-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:19:12.421365 31719 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:19:12.422816 31719 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:19:12.436679 31745 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:19:12.436718 31746 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:19:12.437387 31748 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:19:12.438120 31719 server_base.cc:1034] running on GCE node
I20260620 20:19:12.438724 31719 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:19:12.439731 31719 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:19:12.440958 31719 hybrid_clock.cc:648] HybridClock initialized: now 1781986752440899 us; error 52 us; skew 500 ppm
I20260620 20:19:12.441426 31719 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:19:12 dist-test-slave-2hpg krb5kdc[31098](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986752, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.217.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:19:12.450835 31719 init.cc:377] Logged in from keytab as kudu/127.16.217.1@KRBTEST.COM (short username kudu)
I20260620 20:19:12.452625 31719 webserver.cc:460] Webserver started at http://127.16.217.1:38787/ using document root <none> and password file <none>
I20260620 20:19:12.453362 31719 fs_manager.cc:359] Metadata directory not provided
I20260620 20:19:12.453482 31719 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:19:12.460014 31719 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 20:19:12.463294 31755 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:19:12.464926 31719 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260620 20:19:12.465147 31719 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986536687152-17252-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986536687152-17252-0/minicluster-data/ts-0/wal
uuid: "8400677523bb4227b8e2d7b83e7f45ba"
format_stamp: "Formatted at 2026-06-20 20:19:09 on dist-test-slave-2hpg"
server_key: "1282b3fbdc54571751d10a364a758a0f"
server_key_iv: "b0843de0908b3105cefbbeef1b335a8a"
server_key_version: "encryptionkey@0"
I20260620 20:19:12.466367 31719 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986536687152-17252-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986536687152-17252-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986536687152-17252-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:19:12.486900 31719 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:19:12.491992 31719 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:19:12.492285 31719 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:19:12.493431 31719 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:19:12.495016 31719 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:19:12.495141 31719 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:19:12.495288 31719 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:19:12.495373 31719 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:19:12.531714 31719 rpc_server.cc:307] RPC server started. Bound to: 127.16.217.1:44135
I20260620 20:19:12.531831 31867 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.217.1:44135 every 8 connection(s)
I20260620 20:19:12.533282 31719 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986536687152-17252-0/minicluster-data/ts-0/data/info.pb
I20260620 20:19:12.543685 17252 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu as pid 31719
I20260620 20:19:12.544304 17252 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
/tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986536687152-17252-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986536687152-17252-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986536687152-17252-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986536687152-17252-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.217.2:33683
--local_ip_for_outbound_sockets=127.16.217.2
--tserver_master_addrs=127.16.217.62:36477
--webserver_port=44817
--webserver_interface=127.16.217.2
--keytab_file=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.2.keytab
--principal=kudu/127.16.217.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.217.20:38487
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986536687152-17252-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986536687152-17252-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986536687152-17252-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 20:19:12 dist-test-slave-2hpg krb5kdc[31098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986752, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.217.1@KRBTEST.COM for kudu/127.16.217.62@KRBTEST.COM
I20260620 20:19:12.558279 31871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:19:12.536458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:36477 (local address 127.16.217.1:53259)
0620 20:19:12.537357 (+ 899us) negotiation.cc:107] Waiting for socket to connect
0620 20:19:12.537413 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0620 20:19:12.539197 (+ 1784us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:19:12.540458 (+ 1261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:19:12.540465 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:19:12.540963 (+ 498us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:19:12.541530 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:19:12.541549 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:19:12.543245 (+ 1696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:19:12.543249 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:19:12.544674 (+ 1425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:19:12.544687 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:19:12.546227 (+ 1540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:19:12.546232 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:19:12.546416 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:19:12.547132 (+ 716us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:19:12.547155 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:19:12.549297 (+ 2142us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:19:12.554170 (+ 4873us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:19:12.554176 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:19:12.554188 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:19:12.554596 (+ 408us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:19:12.555112 (+ 516us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:19:12.555115 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:19:12.555117 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:19:12.555310 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:19:12.556275 (+ 965us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:19:12.556281 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:19:12.556576 (+ 295us) client_negotiation.cc:769] Sending connection context
0620 20:19:12.556805 (+ 229us) client_negotiation.cc:240] Negotiation successful
0620 20:19:12.557189 (+ 384us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":591,"thread_start_us":383,"threads_started":1}
I20260620 20:19:12.558856 31870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:19:12.536225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.217.1:53259 (local address 127.16.217.62:36477)
0620 20:19:12.536828 (+ 603us) server_negotiation.cc:206] Beginning negotiation
0620 20:19:12.536840 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 20:19:12.537536 (+ 696us) server_negotiation.cc:407] Connection header received
0620 20:19:12.539692 (+ 2156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:19:12.539706 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:19:12.540021 (+ 315us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:19:12.540315 (+ 294us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:19:12.541749 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:19:12.543085 (+ 1336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:19:12.544826 (+ 1741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:19:12.546005 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:19:12.546218 (+ 213us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:19:12.549578 (+ 3360us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:19:12.549590 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:19:12.549611 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:19:12.549670 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:19:12.554010 (+ 4340us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:19:12.554736 (+ 726us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:19:12.554748 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:19:12.554758 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:19:12.554982 (+ 224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:19:12.555431 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:19:12.555439 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:19:12.555445 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:19:12.556135 (+ 690us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:19:12.556326 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0620 20:19:12.557070 (+ 744us) server_negotiation.cc:299] Negotiation successful
0620 20:19:12.557516 (+ 446us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":422,"thread_start_us":270,"threads_started":1}
I20260620 20:19:12.561787 31868 heartbeater.cc:347] Connected to a master server at 127.16.217.62:36477
I20260620 20:19:12.562142 31868 heartbeater.cc:464] Registering TS with master...
I20260620 20:19:12.562896 31868 heartbeater.cc:511] Master 127.16.217.62:36477 requested a full tablet report, sending...
I20260620 20:19:12.565104 31682 ts_manager.cc:194] Registered new tserver with Master: 8400677523bb4227b8e2d7b83e7f45ba (127.16.217.1:44135)
I20260620 20:19:12.566785 31682 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.217.1@KRBTEST.COM'} at 127.16.217.1:53259
W20260620 20:19:12.875008 31872 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:19:12.875272 31872 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:19:12.875367 31872 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:19:12.882521 31872 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:19:12.882629 31872 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:19:12.882748 31872 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.217.2
I20260620 20:19:12.892504 31872 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.217.20:38487
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986536687152-17252-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986536687152-17252-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.2.keytab
--principal=kudu/127.16.217.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.217.2:33683
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986536687152-17252-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.217.2
--webserver_port=44817
--tserver_master_addrs=127.16.217.62:36477
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.217.2
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986536687152-17252-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:19:12.893694 31872 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:19:12.894953 31872 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:19:12.908521 31877 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:19:12.908494 31878 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:19:12.908459 31880 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:19:12.909325 31872 server_base.cc:1034] running on GCE node
I20260620 20:19:12.910025 31872 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:19:12.910926 31872 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:19:12.912158 31872 hybrid_clock.cc:648] HybridClock initialized: now 1781986752912095 us; error 58 us; skew 500 ppm
I20260620 20:19:12.912472 31872 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:19:12 dist-test-slave-2hpg krb5kdc[31098](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986752, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.217.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:19:12.915783 31872 init.cc:377] Logged in from keytab as kudu/127.16.217.2@KRBTEST.COM (short username kudu)
I20260620 20:19:12.917634 31872 webserver.cc:460] Webserver started at http://127.16.217.2:44817/ using document root <none> and password file <none>
I20260620 20:19:12.918349 31872 fs_manager.cc:359] Metadata directory not provided
I20260620 20:19:12.918464 31872 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:19:12.923723 31872 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 20:19:12.926397 31887 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:19:12.927700 31872 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 20:19:12.927827 31872 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986536687152-17252-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986536687152-17252-0/minicluster-data/ts-1/wal
uuid: "89eac4104e6c4573b766d41f6999395e"
format_stamp: "Formatted at 2026-06-20 20:19:10 on dist-test-slave-2hpg"
server_key: "322fa0b1cbcc51e6e4d913d5dde29c77"
server_key_iv: "12d9c585b627ebd71485d82d57331537"
server_key_version: "encryptionkey@0"
I20260620 20:19:12.928746 31872 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986536687152-17252-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986536687152-17252-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986536687152-17252-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:19:12.944584 31872 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:19:12.948566 31872 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:19:12.948796 31872 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:19:12.949668 31872 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:19:12.950953 31872 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:19:12.951047 31872 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:19:12.951116 31872 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:19:12.951174 31872 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:19:12.986120 31872 rpc_server.cc:307] RPC server started. Bound to: 127.16.217.2:33683
I20260620 20:19:12.986248 31999 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.217.2:33683 every 8 connection(s)
I20260620 20:19:12.987582 31872 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986536687152-17252-0/minicluster-data/ts-1/data/info.pb
I20260620 20:19:12.991096 17252 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu as pid 31872
I20260620 20:19:12.991750 17252 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
/tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986536687152-17252-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986536687152-17252-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986536687152-17252-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986536687152-17252-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.217.3:34191
--local_ip_for_outbound_sockets=127.16.217.3
--tserver_master_addrs=127.16.217.62:36477
--webserver_port=39173
--webserver_interface=127.16.217.3
--keytab_file=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.3.keytab
--principal=kudu/127.16.217.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.217.20:38487
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986536687152-17252-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986536687152-17252-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986536687152-17252-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 20:19:12 dist-test-slave-2hpg krb5kdc[31098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986752, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.217.2@KRBTEST.COM for kudu/127.16.217.62@KRBTEST.COM
I20260620 20:19:13.009120 31870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:19:12.990193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.217.2:45193 (local address 127.16.217.62:36477)
0620 20:19:12.990379 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0620 20:19:12.990390 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 20:19:12.991773 (+ 1383us) server_negotiation.cc:407] Connection header received
0620 20:19:12.993604 (+ 1831us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:19:12.993613 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:19:12.993724 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:19:12.993845 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:19:12.995641 (+ 1796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:19:12.996348 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:19:12.997958 (+ 1610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:19:12.998390 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:19:12.998535 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:19:13.002821 (+ 4286us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:19:13.002826 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:19:13.002829 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:19:13.002863 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:19:13.005467 (+ 2604us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:19:13.006446 (+ 979us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:19:13.006450 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:19:13.006453 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:19:13.006580 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:19:13.007078 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:19:13.007081 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:19:13.007083 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:19:13.007482 (+ 399us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:19:13.007675 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0620 20:19:13.008670 (+ 995us) server_negotiation.cc:299] Negotiation successful
0620 20:19:13.008862 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 20:19:13.010854 32002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:19:12.990725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:36477 (local address 127.16.217.2:45193)
0620 20:19:12.991599 (+ 874us) negotiation.cc:107] Waiting for socket to connect
0620 20:19:12.991650 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 20:19:12.993328 (+ 1678us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:19:12.994105 (+ 777us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:19:12.994113 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:19:12.994852 (+ 739us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:19:12.995428 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:19:12.995445 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:19:12.996525 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:19:12.996530 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:19:12.997782 (+ 1252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:19:12.997793 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:19:12.998615 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:19:12.998622 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:19:12.998930 (+ 308us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:19:12.999628 (+ 698us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:19:12.999661 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:19:13.002600 (+ 2939us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:19:13.005630 (+ 3030us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:19:13.005642 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:19:13.005661 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:19:13.006319 (+ 658us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:19:13.006700 (+ 381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:19:13.006708 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:19:13.006714 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:19:13.006966 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:19:13.007655 (+ 689us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:19:13.007667 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:19:13.008168 (+ 501us) client_negotiation.cc:769] Sending connection context
0620 20:19:13.008506 (+ 338us) client_negotiation.cc:240] Negotiation successful
0620 20:19:13.009182 (+ 676us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":589,"thread_start_us":421,"threads_started":1}
I20260620 20:19:13.012658 32000 heartbeater.cc:347] Connected to a master server at 127.16.217.62:36477
I20260620 20:19:13.013000 32000 heartbeater.cc:464] Registering TS with master...
I20260620 20:19:13.013782 32000 heartbeater.cc:511] Master 127.16.217.62:36477 requested a full tablet report, sending...
I20260620 20:19:13.015329 31682 ts_manager.cc:194] Registered new tserver with Master: 89eac4104e6c4573b766d41f6999395e (127.16.217.2:33683)
I20260620 20:19:13.016197 31682 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.217.2@KRBTEST.COM'} at 127.16.217.2:45193
W20260620 20:19:13.322220 32003 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:19:13.322574 32003 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:19:13.322763 32003 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:19:13.330308 32003 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:19:13.330483 32003 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:19:13.330675 32003 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.217.3
I20260620 20:19:13.340608 32003 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.217.20:38487
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986536687152-17252-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986536687152-17252-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.3.keytab
--principal=kudu/127.16.217.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.217.3:34191
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986536687152-17252-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.217.3
--webserver_port=39173
--tserver_master_addrs=127.16.217.62:36477
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.217.3
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986536687152-17252-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:19:13.342026 32003 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:19:13.343348 32003 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:19:13.356662 32011 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:19:13.356714 32009 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:19:13.356943 32008 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:19:13.357708 32003 server_base.cc:1034] running on GCE node
I20260620 20:19:13.358341 32003 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:19:13.359249 32003 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:19:13.360507 32003 hybrid_clock.cc:648] HybridClock initialized: now 1781986753360451 us; error 71 us; skew 500 ppm
I20260620 20:19:13.360929 32003 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:19:13 dist-test-slave-2hpg krb5kdc[31098](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986753, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.217.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:19:13.364428 32003 init.cc:377] Logged in from keytab as kudu/127.16.217.3@KRBTEST.COM (short username kudu)
I20260620 20:19:13.366161 32003 webserver.cc:460] Webserver started at http://127.16.217.3:39173/ using document root <none> and password file <none>
I20260620 20:19:13.366873 32003 fs_manager.cc:359] Metadata directory not provided
I20260620 20:19:13.367115 32003 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:19:13.373281 32003 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.003s
I20260620 20:19:13.376303 32018 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:19:13.377660 32003 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 20:19:13.377846 32003 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986536687152-17252-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986536687152-17252-0/minicluster-data/ts-2/wal
uuid: "5ee69ada9a214bc8a73ad8e534401c60"
format_stamp: "Formatted at 2026-06-20 20:19:10 on dist-test-slave-2hpg"
server_key: "affbce870a7beb9fcad9c5c2b6384c61"
server_key_iv: "9b939c1a3d3d41a24e1720b1879e1358"
server_key_version: "encryptionkey@0"
I20260620 20:19:13.378973 32003 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986536687152-17252-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986536687152-17252-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986536687152-17252-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:19:13.404603 32003 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:19:13.409262 32003 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:19:13.409583 32003 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:19:13.410645 32003 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:19:13.412006 32003 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:19:13.412142 32003 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:19:13.412288 32003 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:19:13.412384 32003 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:19:13.446882 32003 rpc_server.cc:307] RPC server started. Bound to: 127.16.217.3:34191
I20260620 20:19:13.446977 32130 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.217.3:34191 every 8 connection(s)
I20260620 20:19:13.448441 32003 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986536687152-17252-0/minicluster-data/ts-2/data/info.pb
I20260620 20:19:13.453873 17252 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu as pid 32003
Jun 20 20:19:13 dist-test-slave-2hpg krb5kdc[31098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986753, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.217.3@KRBTEST.COM for kudu/127.16.217.62@KRBTEST.COM
I20260620 20:19:13.472767 32134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:19:13.454932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41016 (local address 127.16.217.62:36477)
0620 20:19:13.455515 (+ 583us) server_negotiation.cc:206] Beginning negotiation
0620 20:19:13.455525 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 20:19:13.456104 (+ 579us) server_negotiation.cc:407] Connection header received
0620 20:19:13.456244 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:19:13.456251 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:19:13.456329 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:19:13.456436 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:19:13.458641 (+ 2205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:19:13.459782 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:19:13.461973 (+ 2191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:19:13.462534 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:19:13.462703 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:19:13.465788 (+ 3085us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:19:13.465794 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:19:13.465797 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:19:13.465843 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:19:13.469445 (+ 3602us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:19:13.470313 (+ 868us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:19:13.470320 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:19:13.470324 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:19:13.470495 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:19:13.470978 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:19:13.470983 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:19:13.470986 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:19:13.471319 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:19:13.471579 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0620 20:19:13.471938 (+ 359us) server_negotiation.cc:299] Negotiation successful
0620 20:19:13.472305 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":419,"spinlock_wait_cycles":5888,"thread_start_us":274,"threads_started":1}
I20260620 20:19:13.473646 31870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:19:13.451295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.217.3:55903 (local address 127.16.217.62:36477)
0620 20:19:13.451511 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0620 20:19:13.451515 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:19:13.452690 (+ 1175us) server_negotiation.cc:407] Connection header received
0620 20:19:13.454640 (+ 1950us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:19:13.454649 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:19:13.454755 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:19:13.454881 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:19:13.456708 (+ 1827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:19:13.457991 (+ 1283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:19:13.460187 (+ 2196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:19:13.461009 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:19:13.461160 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:19:13.465304 (+ 4144us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:19:13.465310 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:19:13.465313 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:19:13.465352 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:19:13.469399 (+ 4047us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:19:13.470373 (+ 974us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:19:13.470379 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:19:13.470383 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:19:13.470532 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:19:13.471108 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:19:13.471114 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:19:13.471118 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:19:13.471419 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:19:13.471603 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0620 20:19:13.472980 (+ 1377us) server_negotiation.cc:299] Negotiation successful
0620 20:19:13.473237 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73,"spinlock_wait_cycles":1920}
I20260620 20:19:13.474763 32133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:19:13.451774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:36477 (local address 127.16.217.3:55903)
0620 20:19:13.452468 (+ 694us) negotiation.cc:107] Waiting for socket to connect
0620 20:19:13.452517 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0620 20:19:13.454351 (+ 1834us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:19:13.455155 (+ 804us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:19:13.455172 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:19:13.455723 (+ 551us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:19:13.456448 (+ 725us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:19:13.456474 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:19:13.458157 (+ 1683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:19:13.458160 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:19:13.460018 (+ 1858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:19:13.460030 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:19:13.461232 (+ 1202us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:19:13.461237 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:19:13.461450 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:19:13.462201 (+ 751us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:19:13.462235 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:19:13.464906 (+ 2671us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:19:13.469578 (+ 4672us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:19:13.469589 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:19:13.469608 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:19:13.470210 (+ 602us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:19:13.470672 (+ 462us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:19:13.470678 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:19:13.470682 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:19:13.470967 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:19:13.471678 (+ 711us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:19:13.471685 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:19:13.472354 (+ 669us) client_negotiation.cc:769] Sending connection context
0620 20:19:13.472997 (+ 643us) client_negotiation.cc:240] Negotiation successful
0620 20:19:13.473570 (+ 573us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":280,"threads_started":1}
I20260620 20:19:13.476333 32131 heartbeater.cc:347] Connected to a master server at 127.16.217.62:36477
I20260620 20:19:13.476900 32131 heartbeater.cc:464] Registering TS with master...
I20260620 20:19:13.477984 32131 heartbeater.cc:511] Master 127.16.217.62:36477 requested a full tablet report, sending...
I20260620 20:19:13.479570 31682 ts_manager.cc:194] Registered new tserver with Master: 5ee69ada9a214bc8a73ad8e534401c60 (127.16.217.3:34191)
I20260620 20:19:13.480439 31682 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.217.3@KRBTEST.COM'} at 127.16.217.3:55903
I20260620 20:19:13.488140 17252 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 20:19:13.496151 31870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:19:13.489417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41018 (local address 127.16.217.62:36477)
0620 20:19:13.489642 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0620 20:19:13.489646 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:19:13.489933 (+ 287us) server_negotiation.cc:407] Connection header received
0620 20:19:13.490244 (+ 311us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:19:13.490251 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:19:13.490404 (+ 153us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:19:13.490549 (+ 145us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 20:19:13.491016 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:19:13.491873 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:19:13.493459 (+ 1586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:19:13.494051 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:19:13.494189 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:19:13.494900 (+ 711us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 20:19:13.495420 (+ 520us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 20:19:13.495538 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0620 20:19:13.495780 (+ 242us) server_negotiation.cc:299] Negotiation successful
0620 20:19:13.495850 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260620 20:19:13.497107 31682 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:41018:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 20:19:13.504546 31682 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 20:19:13.539402 32141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:19:13.516801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.2:33683 (local address 127.0.0.1:54030)
0620 20:19:13.528592 (+ 11791us) negotiation.cc:107] Waiting for socket to connect
0620 20:19:13.528614 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 20:19:13.528742 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:19:13.529987 (+ 1245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:19:13.529994 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:19:13.530021 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:19:13.530551 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:19:13.530571 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:19:13.533445 (+ 2874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:19:13.533449 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:19:13.536384 (+ 2935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:19:13.536400 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:19:13.538375 (+ 1975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:19:13.538380 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:19:13.538571 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:19:13.538634 (+ 63us) client_negotiation.cc:769] Sending connection context
0620 20:19:13.538749 (+ 115us) client_negotiation.cc:240] Negotiation successful
0620 20:19:13.538923 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11640,"mutex_wait_us":222,"thread_start_us":10698,"threads_started":1}
I20260620 20:19:13.540311 32146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:19:13.518072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34620 (local address 127.16.217.3:34191)
0620 20:19:13.525319 (+ 7247us) server_negotiation.cc:206] Beginning negotiation
0620 20:19:13.525332 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 20:19:13.529888 (+ 4556us) server_negotiation.cc:407] Connection header received
0620 20:19:13.529985 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:19:13.529993 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:19:13.530182 (+ 189us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:19:13.530361 (+ 179us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:19:13.531502 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:19:13.534207 (+ 2705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:19:13.537761 (+ 3554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:19:13.539146 (+ 1385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:19:13.539305 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:19:13.539528 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0620 20:19:13.539809 (+ 281us) server_negotiation.cc:299] Negotiation successful
0620 20:19:13.539986 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6999,"thread_start_us":6806,"threads_started":1}
I20260620 20:19:13.540311 32143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:19:13.516849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57852 (local address 127.16.217.1:44135)
0620 20:19:13.529233 (+ 12384us) server_negotiation.cc:206] Beginning negotiation
0620 20:19:13.529246 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 20:19:13.529280 (+ 34us) server_negotiation.cc:407] Connection header received
0620 20:19:13.529402 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:19:13.529412 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:19:13.529698 (+ 286us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:19:13.529915 (+ 217us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:19:13.531213 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:19:13.533140 (+ 1927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:19:13.536597 (+ 3457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:19:13.538148 (+ 1551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:19:13.538294 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:19:13.538537 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0620 20:19:13.539839 (+ 1302us) server_negotiation.cc:299] Negotiation successful
0620 20:19:13.539990 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12216,"thread_start_us":13410,"threads_started":1}
I20260620 20:19:13.540313 32144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:19:13.517257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.3:34191 (local address 127.0.0.1:34620)
0620 20:19:13.529595 (+ 12338us) negotiation.cc:107] Waiting for socket to connect
0620 20:19:13.529622 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0620 20:19:13.529740 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:19:13.530502 (+ 762us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:19:13.530506 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:19:13.530537 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:19:13.531283 (+ 746us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:19:13.531300 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:19:13.534376 (+ 3076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:19:13.534381 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:19:13.537590 (+ 3209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:19:13.537603 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:19:13.539406 (+ 1803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:19:13.539413 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:19:13.539606 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:19:13.539639 (+ 33us) client_negotiation.cc:769] Sending connection context
0620 20:19:13.539739 (+ 100us) client_negotiation.cc:240] Negotiation successful
0620 20:19:13.539882 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12231,"thread_start_us":12064,"threads_started":1}
I20260620 20:19:13.540838 32142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:19:13.516688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.1:44135 (local address 127.0.0.1:57852)
0620 20:19:13.527845 (+ 11157us) negotiation.cc:107] Waiting for socket to connect
0620 20:19:13.527892 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 20:19:13.528067 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:19:13.530172 (+ 2105us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:19:13.530176 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:19:13.530204 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:19:13.531011 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:19:13.531024 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:19:13.533422 (+ 2398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:19:13.533430 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:19:13.536427 (+ 2997us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:19:13.536436 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:19:13.538696 (+ 2260us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:19:13.538703 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:19:13.538875 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:19:13.538904 (+ 29us) client_negotiation.cc:769] Sending connection context
0620 20:19:13.538972 (+ 68us) client_negotiation.cc:240] Negotiation successful
0620 20:19:13.539098 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10921,"mutex_wait_us":364,"thread_start_us":11593,"threads_started":1}
I20260620 20:19:13.541399 32145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:19:13.517538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54030 (local address 127.16.217.2:33683)
0620 20:19:13.529356 (+ 11818us) server_negotiation.cc:206] Beginning negotiation
0620 20:19:13.529365 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 20:19:13.529390 (+ 25us) server_negotiation.cc:407] Connection header received
0620 20:19:13.529484 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:19:13.529492 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:19:13.529734 (+ 242us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:19:13.530660 (+ 926us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:19:13.531598 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:19:13.533299 (+ 1701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:19:13.536588 (+ 3289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:19:13.538148 (+ 1560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:19:13.540645 (+ 2497us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:19:13.540842 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0620 20:19:13.540911 (+ 69us) server_negotiation.cc:299] Negotiation successful
0620 20:19:13.541079 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11628,"thread_start_us":11448,"threads_started":1}
I20260620 20:19:13.544246 31935 tablet_service.cc:1467] Processing CreateTablet for tablet 8b4678944eb04ee2856f77c1278ba65a (DEFAULT_TABLE table=test-table [id=c76eb043d1a0414b88fe7f3c35f3c255]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:19:13.544137 32066 tablet_service.cc:1467] Processing CreateTablet for tablet 8b4678944eb04ee2856f77c1278ba65a (DEFAULT_TABLE table=test-table [id=c76eb043d1a0414b88fe7f3c35f3c255]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:19:13.545539 32066 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8b4678944eb04ee2856f77c1278ba65a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:19:13.545560 31935 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8b4678944eb04ee2856f77c1278ba65a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:19:13.545181 31803 tablet_service.cc:1467] Processing CreateTablet for tablet 8b4678944eb04ee2856f77c1278ba65a (DEFAULT_TABLE table=test-table [id=c76eb043d1a0414b88fe7f3c35f3c255]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:19:13.546432 31803 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8b4678944eb04ee2856f77c1278ba65a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:19:13.559736 32147 tablet_bootstrap.cc:492] T 8b4678944eb04ee2856f77c1278ba65a P 89eac4104e6c4573b766d41f6999395e: Bootstrap starting.
I20260620 20:19:13.561751 32148 tablet_bootstrap.cc:492] T 8b4678944eb04ee2856f77c1278ba65a P 8400677523bb4227b8e2d7b83e7f45ba: Bootstrap starting.
I20260620 20:19:13.563448 32149 tablet_bootstrap.cc:492] T 8b4678944eb04ee2856f77c1278ba65a P 5ee69ada9a214bc8a73ad8e534401c60: Bootstrap starting.
I20260620 20:19:13.563815 32147 tablet_bootstrap.cc:654] T 8b4678944eb04ee2856f77c1278ba65a P 89eac4104e6c4573b766d41f6999395e: Neither blocks nor log segments found. Creating new log.
I20260620 20:19:13.565152 32147 log.cc:824] T 8b4678944eb04ee2856f77c1278ba65a P 89eac4104e6c4573b766d41f6999395e: Log is configured to *not* fsync() on all Append() calls
I20260620 20:19:13.565589 32148 tablet_bootstrap.cc:654] T 8b4678944eb04ee2856f77c1278ba65a P 8400677523bb4227b8e2d7b83e7f45ba: Neither blocks nor log segments found. Creating new log.
I20260620 20:19:13.566313 32149 tablet_bootstrap.cc:654] T 8b4678944eb04ee2856f77c1278ba65a P 5ee69ada9a214bc8a73ad8e534401c60: Neither blocks nor log segments found. Creating new log.
I20260620 20:19:13.566845 32148 log.cc:824] T 8b4678944eb04ee2856f77c1278ba65a P 8400677523bb4227b8e2d7b83e7f45ba: Log is configured to *not* fsync() on all Append() calls
I20260620 20:19:13.567488 32149 log.cc:824] T 8b4678944eb04ee2856f77c1278ba65a P 5ee69ada9a214bc8a73ad8e534401c60: Log is configured to *not* fsync() on all Append() calls
I20260620 20:19:13.568833 32147 tablet_bootstrap.cc:492] T 8b4678944eb04ee2856f77c1278ba65a P 89eac4104e6c4573b766d41f6999395e: No bootstrap required, opened a new log
I20260620 20:19:13.569166 32147 ts_tablet_manager.cc:1397] T 8b4678944eb04ee2856f77c1278ba65a P 89eac4104e6c4573b766d41f6999395e: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.004s
I20260620 20:19:13.569844 31868 heartbeater.cc:503] Master 127.16.217.62:36477 was elected leader, sending a full tablet report...
I20260620 20:19:13.571192 32148 tablet_bootstrap.cc:492] T 8b4678944eb04ee2856f77c1278ba65a P 8400677523bb4227b8e2d7b83e7f45ba: No bootstrap required, opened a new log
I20260620 20:19:13.571516 32148 ts_tablet_manager.cc:1397] T 8b4678944eb04ee2856f77c1278ba65a P 8400677523bb4227b8e2d7b83e7f45ba: Time spent bootstrapping tablet: real 0.010s user 0.009s sys 0.000s
I20260620 20:19:13.571568 32149 tablet_bootstrap.cc:492] T 8b4678944eb04ee2856f77c1278ba65a P 5ee69ada9a214bc8a73ad8e534401c60: No bootstrap required, opened a new log
I20260620 20:19:13.571869 32149 ts_tablet_manager.cc:1397] T 8b4678944eb04ee2856f77c1278ba65a P 5ee69ada9a214bc8a73ad8e534401c60: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260620 20:19:13.577986 32147 raft_consensus.cc:348] T 8b4678944eb04ee2856f77c1278ba65a P 89eac4104e6c4573b766d41f6999395e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8400677523bb4227b8e2d7b83e7f45ba" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 44135 } } peers { permanent_uuid: "89eac4104e6c4573b766d41f6999395e" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 33683 } } peers { permanent_uuid: "5ee69ada9a214bc8a73ad8e534401c60" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 34191 } }
I20260620 20:19:13.578474 32147 raft_consensus.cc:374] T 8b4678944eb04ee2856f77c1278ba65a P 89eac4104e6c4573b766d41f6999395e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:19:13.578614 32147 raft_consensus.cc:729] T 8b4678944eb04ee2856f77c1278ba65a P 89eac4104e6c4573b766d41f6999395e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89eac4104e6c4573b766d41f6999395e, State: Initialized, Role: FOLLOWER
I20260620 20:19:13.579253 32147 consensus_queue.cc:260] T 8b4678944eb04ee2856f77c1278ba65a P 89eac4104e6c4573b766d41f6999395e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8400677523bb4227b8e2d7b83e7f45ba" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 44135 } } peers { permanent_uuid: "89eac4104e6c4573b766d41f6999395e" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 33683 } } peers { permanent_uuid: "5ee69ada9a214bc8a73ad8e534401c60" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 34191 } }
I20260620 20:19:13.580349 32000 heartbeater.cc:503] Master 127.16.217.62:36477 was elected leader, sending a full tablet report...
I20260620 20:19:13.580848 32147 ts_tablet_manager.cc:1428] T 8b4678944eb04ee2856f77c1278ba65a P 89eac4104e6c4573b766d41f6999395e: Time spent starting tablet: real 0.012s user 0.010s sys 0.001s
I20260620 20:19:13.582412 32149 raft_consensus.cc:348] T 8b4678944eb04ee2856f77c1278ba65a P 5ee69ada9a214bc8a73ad8e534401c60 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8400677523bb4227b8e2d7b83e7f45ba" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 44135 } } peers { permanent_uuid: "89eac4104e6c4573b766d41f6999395e" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 33683 } } peers { permanent_uuid: "5ee69ada9a214bc8a73ad8e534401c60" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 34191 } }
I20260620 20:19:13.582880 32149 raft_consensus.cc:374] T 8b4678944eb04ee2856f77c1278ba65a P 5ee69ada9a214bc8a73ad8e534401c60 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:19:13.582993 32149 raft_consensus.cc:729] T 8b4678944eb04ee2856f77c1278ba65a P 5ee69ada9a214bc8a73ad8e534401c60 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ee69ada9a214bc8a73ad8e534401c60, State: Initialized, Role: FOLLOWER
I20260620 20:19:13.582852 32148 raft_consensus.cc:348] T 8b4678944eb04ee2856f77c1278ba65a P 8400677523bb4227b8e2d7b83e7f45ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8400677523bb4227b8e2d7b83e7f45ba" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 44135 } } peers { permanent_uuid: "89eac4104e6c4573b766d41f6999395e" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 33683 } } peers { permanent_uuid: "5ee69ada9a214bc8a73ad8e534401c60" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 34191 } }
I20260620 20:19:13.583371 32148 raft_consensus.cc:374] T 8b4678944eb04ee2856f77c1278ba65a P 8400677523bb4227b8e2d7b83e7f45ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:19:13.583498 32148 raft_consensus.cc:729] T 8b4678944eb04ee2856f77c1278ba65a P 8400677523bb4227b8e2d7b83e7f45ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8400677523bb4227b8e2d7b83e7f45ba, State: Initialized, Role: FOLLOWER
I20260620 20:19:13.583634 32149 consensus_queue.cc:260] T 8b4678944eb04ee2856f77c1278ba65a P 5ee69ada9a214bc8a73ad8e534401c60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8400677523bb4227b8e2d7b83e7f45ba" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 44135 } } peers { permanent_uuid: "89eac4104e6c4573b766d41f6999395e" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 33683 } } peers { permanent_uuid: "5ee69ada9a214bc8a73ad8e534401c60" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 34191 } }
I20260620 20:19:13.584089 32148 consensus_queue.cc:260] T 8b4678944eb04ee2856f77c1278ba65a P 8400677523bb4227b8e2d7b83e7f45ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8400677523bb4227b8e2d7b83e7f45ba" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 44135 } } peers { permanent_uuid: "89eac4104e6c4573b766d41f6999395e" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 33683 } } peers { permanent_uuid: "5ee69ada9a214bc8a73ad8e534401c60" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 34191 } }
I20260620 20:19:13.587041 32148 ts_tablet_manager.cc:1428] T 8b4678944eb04ee2856f77c1278ba65a P 8400677523bb4227b8e2d7b83e7f45ba: Time spent starting tablet: real 0.015s user 0.011s sys 0.003s
I20260620 20:19:13.589852 32131 heartbeater.cc:503] Master 127.16.217.62:36477 was elected leader, sending a full tablet report...
I20260620 20:19:13.590548 32149 ts_tablet_manager.cc:1428] T 8b4678944eb04ee2856f77c1278ba65a P 5ee69ada9a214bc8a73ad8e534401c60: Time spent starting tablet: real 0.019s user 0.008s sys 0.010s
W20260620 20:19:13.700184 32132 tablet.cc:2369] T 8b4678944eb04ee2856f77c1278ba65a P 5ee69ada9a214bc8a73ad8e534401c60: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 20:19:13.710835 32153 raft_consensus.cc:482] T 8b4678944eb04ee2856f77c1278ba65a P 89eac4104e6c4573b766d41f6999395e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 20:19:13.711146 32153 raft_consensus.cc:504] T 8b4678944eb04ee2856f77c1278ba65a P 89eac4104e6c4573b766d41f6999395e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8400677523bb4227b8e2d7b83e7f45ba" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 44135 } } peers { permanent_uuid: "89eac4104e6c4573b766d41f6999395e" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 33683 } } peers { permanent_uuid: "5ee69ada9a214bc8a73ad8e534401c60" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 34191 } }
I20260620 20:19:13.712802 32153 leader_election.cc:288] T 8b4678944eb04ee2856f77c1278ba65a P 89eac4104e6c4573b766d41f6999395e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8400677523bb4227b8e2d7b83e7f45ba (127.16.217.1:44135), 5ee69ada9a214bc8a73ad8e534401c60 (127.16.217.3:34191)
I20260620 20:19:13.722713 32146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:19:13.713246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.217.2:60145 (local address 127.16.217.3:34191)
0620 20:19:13.713449 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0620 20:19:13.713456 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:19:13.714516 (+ 1060us) server_negotiation.cc:407] Connection header received
0620 20:19:13.714723 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:19:13.714728 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:19:13.714805 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:19:13.714908 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:19:13.715786 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:19:13.716929 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:19:13.720096 (+ 3167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:19:13.721354 (+ 1258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:19:13.721521 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:19:13.721595 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 20:19:13.722320 (+ 725us) server_negotiation.cc:299] Negotiation successful
0620 20:19:13.722456 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 20:19:13.722735 32143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:19:13.713291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.217.2:59153 (local address 127.16.217.1:44135)
0620 20:19:13.713494 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0620 20:19:13.713501 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:19:13.714522 (+ 1021us) server_negotiation.cc:407] Connection header received
0620 20:19:13.714723 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:19:13.714728 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:19:13.714805 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:19:13.714908 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:19:13.715866 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:19:13.716929 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:19:13.720093 (+ 3164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:19:13.721354 (+ 1261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:19:13.721521 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:19:13.721595 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 20:19:13.722320 (+ 725us) server_negotiation.cc:299] Negotiation successful
0620 20:19:13.722456 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260620 20:19:13.723093 32157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:19:13.713239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.1:44135 (local address 127.16.217.2:59153)
0620 20:19:13.714363 (+ 1124us) negotiation.cc:107] Waiting for socket to connect
0620 20:19:13.714387 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 20:19:13.714549 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:19:13.714989 (+ 440us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:19:13.714996 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:19:13.715109 (+ 113us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:19:13.715683 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:19:13.715699 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:19:13.717102 (+ 1403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:19:13.717107 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:19:13.719904 (+ 2797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:19:13.719921 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:19:13.721673 (+ 1752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:19:13.721680 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:19:13.722071 (+ 391us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:19:13.722174 (+ 103us) client_negotiation.cc:769] Sending connection context
0620 20:19:13.722274 (+ 100us) client_negotiation.cc:240] Negotiation successful
0620 20:19:13.722491 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":987,"mutex_wait_us":359,"spinlock_wait_cycles":27392,"thread_start_us":552,"threads_started":1}
I20260620 20:19:13.723137 32156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:19:13.713240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.3:34191 (local address 127.16.217.2:60145)
0620 20:19:13.714405 (+ 1165us) negotiation.cc:107] Waiting for socket to connect
0620 20:19:13.714423 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 20:19:13.714598 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:19:13.714989 (+ 391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:19:13.714996 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:19:13.715031 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:19:13.715531 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:19:13.715590 (+ 59us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:19:13.717100 (+ 1510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:19:13.717107 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:19:13.719909 (+ 2802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:19:13.719921 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:19:13.721659 (+ 1738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:19:13.721666 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:19:13.722067 (+ 401us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:19:13.722164 (+ 97us) client_negotiation.cc:769] Sending connection context
0620 20:19:13.722274 (+ 110us) client_negotiation.cc:240] Negotiation successful
0620 20:19:13.722491 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1049,"mutex_wait_us":343,"spinlock_wait_cycles":39296,"thread_start_us":557,"threads_started":1}
I20260620 20:19:13.724330 31823 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b4678944eb04ee2856f77c1278ba65a" candidate_uuid: "89eac4104e6c4573b766d41f6999395e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8400677523bb4227b8e2d7b83e7f45ba" is_pre_election: true
I20260620 20:19:13.724330 32086 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b4678944eb04ee2856f77c1278ba65a" candidate_uuid: "89eac4104e6c4573b766d41f6999395e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ee69ada9a214bc8a73ad8e534401c60" is_pre_election: true
I20260620 20:19:13.724925 32086 raft_consensus.cc:2454] T 8b4678944eb04ee2856f77c1278ba65a P 5ee69ada9a214bc8a73ad8e534401c60 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89eac4104e6c4573b766d41f6999395e in term 0.
I20260620 20:19:13.724925 31823 raft_consensus.cc:2454] T 8b4678944eb04ee2856f77c1278ba65a P 8400677523bb4227b8e2d7b83e7f45ba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89eac4104e6c4573b766d41f6999395e in term 0.
I20260620 20:19:13.725680 31891 leader_election.cc:302] T 8b4678944eb04ee2856f77c1278ba65a P 89eac4104e6c4573b766d41f6999395e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ee69ada9a214bc8a73ad8e534401c60, 89eac4104e6c4573b766d41f6999395e; no voters:
I20260620 20:19:13.726140 32153 raft_consensus.cc:2781] T 8b4678944eb04ee2856f77c1278ba65a P 89eac4104e6c4573b766d41f6999395e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 20:19:13.726305 32153 raft_consensus.cc:482] T 8b4678944eb04ee2856f77c1278ba65a P 89eac4104e6c4573b766d41f6999395e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 20:19:13.726419 32153 raft_consensus.cc:3037] T 8b4678944eb04ee2856f77c1278ba65a P 89eac4104e6c4573b766d41f6999395e [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:19:13.728672 32153 raft_consensus.cc:504] T 8b4678944eb04ee2856f77c1278ba65a P 89eac4104e6c4573b766d41f6999395e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8400677523bb4227b8e2d7b83e7f45ba" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 44135 } } peers { permanent_uuid: "89eac4104e6c4573b766d41f6999395e" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 33683 } } peers { permanent_uuid: "5ee69ada9a214bc8a73ad8e534401c60" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 34191 } }
I20260620 20:19:13.729279 32153 leader_election.cc:288] T 8b4678944eb04ee2856f77c1278ba65a P 89eac4104e6c4573b766d41f6999395e [CANDIDATE]: Term 1 election: Requested vote from peers 8400677523bb4227b8e2d7b83e7f45ba (127.16.217.1:44135), 5ee69ada9a214bc8a73ad8e534401c60 (127.16.217.3:34191)
I20260620 20:19:13.729719 31823 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b4678944eb04ee2856f77c1278ba65a" candidate_uuid: "89eac4104e6c4573b766d41f6999395e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8400677523bb4227b8e2d7b83e7f45ba"
I20260620 20:19:13.729723 32086 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b4678944eb04ee2856f77c1278ba65a" candidate_uuid: "89eac4104e6c4573b766d41f6999395e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ee69ada9a214bc8a73ad8e534401c60"
I20260620 20:19:13.729964 31823 raft_consensus.cc:3037] T 8b4678944eb04ee2856f77c1278ba65a P 8400677523bb4227b8e2d7b83e7f45ba [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:19:13.729964 32086 raft_consensus.cc:3037] T 8b4678944eb04ee2856f77c1278ba65a P 5ee69ada9a214bc8a73ad8e534401c60 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:19:13.732251 32086 raft_consensus.cc:2454] T 8b4678944eb04ee2856f77c1278ba65a P 5ee69ada9a214bc8a73ad8e534401c60 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89eac4104e6c4573b766d41f6999395e in term 1.
I20260620 20:19:13.732285 31823 raft_consensus.cc:2454] T 8b4678944eb04ee2856f77c1278ba65a P 8400677523bb4227b8e2d7b83e7f45ba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89eac4104e6c4573b766d41f6999395e in term 1.
I20260620 20:19:13.732772 31890 leader_election.cc:302] T 8b4678944eb04ee2856f77c1278ba65a P 89eac4104e6c4573b766d41f6999395e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8400677523bb4227b8e2d7b83e7f45ba, 89eac4104e6c4573b766d41f6999395e; no voters:
I20260620 20:19:13.733067 32153 raft_consensus.cc:2781] T 8b4678944eb04ee2856f77c1278ba65a P 89eac4104e6c4573b766d41f6999395e [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:19:13.733502 32153 raft_consensus.cc:686] T 8b4678944eb04ee2856f77c1278ba65a P 89eac4104e6c4573b766d41f6999395e [term 1 LEADER]: Becoming Leader. State: Replica: 89eac4104e6c4573b766d41f6999395e, State: Running, Role: LEADER
I20260620 20:19:13.734050 32153 consensus_queue.cc:237] T 8b4678944eb04ee2856f77c1278ba65a P 89eac4104e6c4573b766d41f6999395e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8400677523bb4227b8e2d7b83e7f45ba" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 44135 } } peers { permanent_uuid: "89eac4104e6c4573b766d41f6999395e" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 33683 } } peers { permanent_uuid: "5ee69ada9a214bc8a73ad8e534401c60" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 34191 } }
I20260620 20:19:13.738530 31682 catalog_manager.cc:5495] T 8b4678944eb04ee2856f77c1278ba65a P 89eac4104e6c4573b766d41f6999395e reported cstate change: term changed from 0 to 1, leader changed from <none> to 89eac4104e6c4573b766d41f6999395e (127.16.217.2). New cstate: current_term: 1 leader_uuid: "89eac4104e6c4573b766d41f6999395e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8400677523bb4227b8e2d7b83e7f45ba" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 44135 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "89eac4104e6c4573b766d41f6999395e" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 33683 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5ee69ada9a214bc8a73ad8e534401c60" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 34191 } health_report { overall_health: UNKNOWN } } }
W20260620 20:19:13.739535 32001 tablet.cc:2369] T 8b4678944eb04ee2856f77c1278ba65a P 89eac4104e6c4573b766d41f6999395e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 20:19:13.777510 32145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:19:13.772044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54038 (local address 127.16.217.2:33683)
0620 20:19:13.772280 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0620 20:19:13.772284 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:19:13.772302 (+ 18us) server_negotiation.cc:407] Connection header received
0620 20:19:13.772394 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:19:13.772398 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:19:13.772483 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:19:13.772611 (+ 128us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 20:19:13.773020 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:19:13.773755 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:19:13.775189 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:19:13.775720 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:19:13.775864 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:19:13.776316 (+ 452us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 20:19:13.776885 (+ 569us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 20:19:13.776999 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0620 20:19:13.777142 (+ 143us) server_negotiation.cc:299] Negotiation successful
0620 20:19:13.777205 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
W20260620 20:19:13.785712 31869 tablet.cc:2369] T 8b4678944eb04ee2856f77c1278ba65a P 8400677523bb4227b8e2d7b83e7f45ba: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 20:19:13.785930 31823 raft_consensus.cc:1261] T 8b4678944eb04ee2856f77c1278ba65a P 8400677523bb4227b8e2d7b83e7f45ba [term 1 FOLLOWER]: Refusing update from remote peer 89eac4104e6c4573b766d41f6999395e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:19:13.785925 32086 raft_consensus.cc:1261] T 8b4678944eb04ee2856f77c1278ba65a P 5ee69ada9a214bc8a73ad8e534401c60 [term 1 FOLLOWER]: Refusing update from remote peer 89eac4104e6c4573b766d41f6999395e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:19:13.786890 32153 consensus_queue.cc:1035] T 8b4678944eb04ee2856f77c1278ba65a P 89eac4104e6c4573b766d41f6999395e [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ee69ada9a214bc8a73ad8e534401c60" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 34191 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:19:13.787181 32158 consensus_queue.cc:1035] T 8b4678944eb04ee2856f77c1278ba65a P 89eac4104e6c4573b766d41f6999395e [LEADER]: Connected to new peer: Peer: permanent_uuid: "8400677523bb4227b8e2d7b83e7f45ba" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 44135 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:19:13.798681 32161 mvcc.cc:205] Tried to move back new op lower bound from 7299017743496237056 to 7299017743298686976. Current Snapshot: MvccSnapshot[applied={T|T < 7299017743496237056}]
I20260620 20:19:13.799892 32163 mvcc.cc:205] Tried to move back new op lower bound from 7299017743496237056 to 7299017743298686976. Current Snapshot: MvccSnapshot[applied={T|T < 7299017743496237056}]
I20260620 20:19:13.812618 31681 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:41018:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 20:19:13.812855 31681 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:41018:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 20:19:13.816336 31681 catalog_manager.cc:5782] T 00000000000000000000000000000000 P becbda298fcf43ab9c61810d35e9764a: Sending DeleteTablet for 3 replicas of tablet 8b4678944eb04ee2856f77c1278ba65a
I20260620 20:19:13.817436 31935 tablet_service.cc:1514] Processing DeleteTablet for tablet 8b4678944eb04ee2856f77c1278ba65a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:19:13 UTC) from {username='kudu', principal='kudu/127.16.217.62@KRBTEST.COM'} at 127.0.0.1:54030
I20260620 20:19:13.817484 32066 tablet_service.cc:1514] Processing DeleteTablet for tablet 8b4678944eb04ee2856f77c1278ba65a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:19:13 UTC) from {username='kudu', principal='kudu/127.16.217.62@KRBTEST.COM'} at 127.0.0.1:34620
I20260620 20:19:13.818044 17252 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu pid 31719
I20260620 20:19:13.818174 31803 tablet_service.cc:1514] Processing DeleteTablet for tablet 8b4678944eb04ee2856f77c1278ba65a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:19:13 UTC) from {username='kudu', principal='kudu/127.16.217.62@KRBTEST.COM'} at 127.0.0.1:57852
I20260620 20:19:13.826368 32173 tablet_replica.cc:330] T 8b4678944eb04ee2856f77c1278ba65a P 8400677523bb4227b8e2d7b83e7f45ba: stopping tablet replica
I20260620 20:19:13.826985 32173 raft_consensus.cc:2229] T 8b4678944eb04ee2856f77c1278ba65a P 8400677523bb4227b8e2d7b83e7f45ba [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 20:19:13.827724 32173 raft_consensus.cc:2258] T 8b4678944eb04ee2856f77c1278ba65a P 8400677523bb4227b8e2d7b83e7f45ba [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:19:13.831796 32172 tablet_replica.cc:330] T 8b4678944eb04ee2856f77c1278ba65a P 5ee69ada9a214bc8a73ad8e534401c60: stopping tablet replica
I20260620 20:19:13.832372 32172 raft_consensus.cc:2229] T 8b4678944eb04ee2856f77c1278ba65a P 5ee69ada9a214bc8a73ad8e534401c60 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 20:19:13.832893 32172 raft_consensus.cc:2258] T 8b4678944eb04ee2856f77c1278ba65a P 5ee69ada9a214bc8a73ad8e534401c60 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:19:13.833734 32171 tablet_replica.cc:330] T 8b4678944eb04ee2856f77c1278ba65a P 89eac4104e6c4573b766d41f6999395e: stopping tablet replica
I20260620 20:19:13.834200 32171 raft_consensus.cc:2229] T 8b4678944eb04ee2856f77c1278ba65a P 89eac4104e6c4573b766d41f6999395e [term 1 LEADER]: Raft consensus shutting down.
I20260620 20:19:13.834836 32171 raft_consensus.cc:2258] T 8b4678944eb04ee2856f77c1278ba65a P 89eac4104e6c4573b766d41f6999395e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:19:13.835039 32172 ts_tablet_manager.cc:1905] T 8b4678944eb04ee2856f77c1278ba65a P 5ee69ada9a214bc8a73ad8e534401c60: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:19:13.836825 32173 ts_tablet_manager.cc:1905] T 8b4678944eb04ee2856f77c1278ba65a P 8400677523bb4227b8e2d7b83e7f45ba: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:19:13.842532 32143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:19:13.819883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57854 (local address 127.16.217.1:44135)
0620 20:19:13.820115 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0620 20:19:13.820124 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 20:19:13.820146 (+ 22us) server_negotiation.cc:407] Connection header received
0620 20:19:13.820216 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:19:13.820224 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:19:13.820344 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:19:13.820459 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:19:13.822938 (+ 2479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:19:13.824494 (+ 1556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:19:13.827143 (+ 2649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:19:13.828055 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:19:13.828467 (+ 412us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:19:13.835317 (+ 6850us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:19:13.835335 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:19:13.835347 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:19:13.835388 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:19:13.838417 (+ 3029us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:19:13.839300 (+ 883us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:19:13.839308 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:19:13.839311 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:19:13.839473 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:19:13.839887 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:19:13.839892 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:19:13.839895 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:19:13.840338 (+ 443us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:19:13.840630 (+ 292us) server_negotiation.cc:1035] Waiting for connection context
0620 20:19:13.841829 (+ 1199us) server_negotiation.cc:299] Negotiation successful
0620 20:19:13.842094 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260620 20:19:13.843516 31863 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:19:13.845067 32171 ts_tablet_manager.cc:1905] T 8b4678944eb04ee2856f77c1278ba65a P 89eac4104e6c4573b766d41f6999395e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:19:13.845412 32172 ts_tablet_manager.cc:1918] T 8b4678944eb04ee2856f77c1278ba65a P 5ee69ada9a214bc8a73ad8e534401c60: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 20:19:13.845655 32172 log.cc:1196] T 8b4678944eb04ee2856f77c1278ba65a P 5ee69ada9a214bc8a73ad8e534401c60: Deleting WAL directory at /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986536687152-17252-0/minicluster-data/ts-2/wal/wals/8b4678944eb04ee2856f77c1278ba65a
I20260620 20:19:13.846310 32172 ts_tablet_manager.cc:1939] T 8b4678944eb04ee2856f77c1278ba65a P 5ee69ada9a214bc8a73ad8e534401c60: Deleting consensus metadata
I20260620 20:19:13.848199 31669 catalog_manager.cc:4841] TS 5ee69ada9a214bc8a73ad8e534401c60 (127.16.217.3:34191): tablet 8b4678944eb04ee2856f77c1278ba65a (table test-table [id=c76eb043d1a0414b88fe7f3c35f3c255]) successfully deleted
I20260620 20:19:13.853287 32171 ts_tablet_manager.cc:1918] T 8b4678944eb04ee2856f77c1278ba65a P 89eac4104e6c4573b766d41f6999395e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 20:19:13.853470 32171 log.cc:1196] T 8b4678944eb04ee2856f77c1278ba65a P 89eac4104e6c4573b766d41f6999395e: Deleting WAL directory at /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986536687152-17252-0/minicluster-data/ts-1/wal/wals/8b4678944eb04ee2856f77c1278ba65a
I20260620 20:19:13.854035 32171 ts_tablet_manager.cc:1939] T 8b4678944eb04ee2856f77c1278ba65a P 89eac4104e6c4573b766d41f6999395e: Deleting consensus metadata
I20260620 20:19:13.855410 31666 catalog_manager.cc:4841] TS 89eac4104e6c4573b766d41f6999395e (127.16.217.2:33683): tablet 8b4678944eb04ee2856f77c1278ba65a (table test-table [id=c76eb043d1a0414b88fe7f3c35f3c255]) successfully deleted
I20260620 20:19:13.956763 32173 ts_tablet_manager.cc:1918] T 8b4678944eb04ee2856f77c1278ba65a P 8400677523bb4227b8e2d7b83e7f45ba: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 20:19:13.957036 32173 log.cc:1196] T 8b4678944eb04ee2856f77c1278ba65a P 8400677523bb4227b8e2d7b83e7f45ba: Deleting WAL directory at /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986536687152-17252-0/minicluster-data/ts-0/wal/wals/8b4678944eb04ee2856f77c1278ba65a
I20260620 20:19:13.957371 17252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu with pid 31719
W20260620 20:19:13.975976 31668 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.16.217.1:44135 (error 108)
I20260620 20:19:13.976234 17252 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu pid 31872
I20260620 20:19:13.977942 32142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:19:13.977425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.1:44135 (local address 127.0.0.1:57860)
0620 20:19:13.977664 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0620 20:19:13.977755 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.217.1:44135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123}
W20260620 20:19:13.978281 31668 catalog_manager.cc:4584] TS 8400677523bb4227b8e2d7b83e7f45ba (127.16.217.1:44135): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8b4678944eb04ee2856f77c1278ba65a: Network error: Client connection negotiation failed: client connection to 127.16.217.1:44135: connect: Connection refused (error 111)
I20260620 20:19:13.993655 32145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:19:13.977470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54042 (local address 127.16.217.2:33683)
0620 20:19:13.977688 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0620 20:19:13.977695 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:19:13.977714 (+ 19us) server_negotiation.cc:407] Connection header received
0620 20:19:13.977784 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:19:13.977790 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:19:13.977907 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:19:13.978016 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:19:13.979478 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:19:13.980721 (+ 1243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:19:13.982113 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:19:13.982871 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:19:13.983025 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:19:13.986188 (+ 3163us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:19:13.986200 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:19:13.986217 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:19:13.986266 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:19:13.990529 (+ 4263us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:19:13.991395 (+ 866us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:19:13.991403 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:19:13.991407 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:19:13.991592 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:19:13.991992 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:19:13.991997 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:19:13.992001 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:19:13.992452 (+ 451us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:19:13.992782 (+ 330us) server_negotiation.cc:1035] Waiting for connection context
0620 20:19:13.993053 (+ 271us) server_negotiation.cc:299] Negotiation successful
0620 20:19:13.993274 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260620 20:19:13.994441 31995 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:19:14.032064 32142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:19:14.031560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.1:44135 (local address 127.0.0.1:57866)
0620 20:19:14.031818 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0620 20:19:14.031916 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.217.1:44135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118}
I20260620 20:19:14.033228 32142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:19:14.032869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.1:44135 (local address 127.0.0.1:57868)
0620 20:19:14.033015 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0620 20:19:14.033093 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.217.1:44135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260620 20:19:14.094105 32142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:19:14.093461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.1:44135 (local address 127.0.0.1:57878)
0620 20:19:14.093743 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0620 20:19:14.093932 (+ 189us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.217.1:44135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136}
I20260620 20:19:14.095337 32142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:19:14.094935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.1:44135 (local address 127.0.0.1:57884)
0620 20:19:14.095139 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0620 20:19:14.095220 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.217.1:44135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260620 20:19:14.122018 17252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu with pid 31872
I20260620 20:19:14.141728 17252 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu pid 32003
I20260620 20:19:14.155248 32146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:19:14.142523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34628 (local address 127.16.217.3:34191)
0620 20:19:14.142721 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0620 20:19:14.142725 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:19:14.142744 (+ 19us) server_negotiation.cc:407] Connection header received
0620 20:19:14.142852 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:19:14.142857 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:19:14.142921 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:19:14.143005 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:19:14.144480 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:19:14.145196 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:19:14.146678 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:19:14.147062 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:19:14.147174 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:19:14.149804 (+ 2630us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:19:14.149817 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:19:14.149835 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:19:14.149926 (+ 91us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:19:14.152631 (+ 2705us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:19:14.153274 (+ 643us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:19:14.153281 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:19:14.153283 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:19:14.153419 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:19:14.153745 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:19:14.153748 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:19:14.153750 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:19:14.154202 (+ 452us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:19:14.154505 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0620 20:19:14.154702 (+ 197us) server_negotiation.cc:299] Negotiation successful
0620 20:19:14.154862 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260620 20:19:14.155908 32126 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:19:14.175794 32142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:19:14.175329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.1:44135 (local address 127.0.0.1:57896)
0620 20:19:14.175544 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0620 20:19:14.175627 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.217.1:44135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260620 20:19:14.176846 32142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:19:14.176465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.1:44135 (local address 127.0.0.1:57898)
0620 20:19:14.176661 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0620 20:19:14.176737 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.217.1:44135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
I20260620 20:19:14.243209 17252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu with pid 32003
I20260620 20:19:14.259835 17252 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu pid 31649
I20260620 20:19:14.261029 31712 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:19:14.377703 32142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:19:14.376894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.1:44135 (local address 127.0.0.1:57904)
0620 20:19:14.377115 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0620 20:19:14.377219 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.217.1:44135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260620 20:19:14.377724 17252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu with pid 31649
I20260620 20:19:14.402321 17252 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
/tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986536687152-17252-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986536687152-17252-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986536687152-17252-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986536687152-17252-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.217.62:36477
--webserver_interface=127.16.217.62
--webserver_port=36691
--keytab_file=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.62.keytab
--principal=kudu/127.16.217.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.217.20:38487
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.217.62:36477
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986536687152-17252-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986536687152-17252-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986536687152-17252-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:19:14.737553 32179 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:19:14.737841 32179 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:19:14.738045 32179 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:19:14.745460 32179 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:19:14.745612 32179 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:19:14.745712 32179 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:19:14.745816 32179 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:19:14.745922 32179 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:19:14.756258 32179 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.217.20:38487
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986536687152-17252-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986536687152-17252-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.217.62:36477
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.62.keytab
--principal=kudu/127.16.217.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.217.62:36477
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986536687152-17252-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.217.62
--webserver_port=36691
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986536687152-17252-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:19:14.757680 32179 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:19:14.759069 32179 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:19:14.770500 32184 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:19:14.770462 32185 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:19:14.770525 32187 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:19:14.771512 32179 server_base.cc:1034] running on GCE node
I20260620 20:19:14.772275 32179 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:19:14.773624 32179 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:19:14.774832 32179 hybrid_clock.cc:648] HybridClock initialized: now 1781986754774762 us; error 55 us; skew 500 ppm
I20260620 20:19:14.775166 32179 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:19:14 dist-test-slave-2hpg krb5kdc[31098](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986754, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.217.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:19:14.778369 32179 init.cc:377] Logged in from keytab as kudu/127.16.217.62@KRBTEST.COM (short username kudu)
I20260620 20:19:14.779932 32179 webserver.cc:460] Webserver started at http://127.16.217.62:36691/ using document root <none> and password file <none>
I20260620 20:19:14.780581 32179 fs_manager.cc:359] Metadata directory not provided
I20260620 20:19:14.780730 32179 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:19:14.786041 32179 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 20:19:14.788734 32194 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:19:14.790096 32179 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 20:19:14.790266 32179 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986536687152-17252-0/minicluster-data/master-0/data,/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986536687152-17252-0/minicluster-data/master-0/wal
uuid: "becbda298fcf43ab9c61810d35e9764a"
format_stamp: "Formatted at 2026-06-20 20:19:08 on dist-test-slave-2hpg"
server_key: "c6d40976d637a0ffe5a5a7c91c0ba6c9"
server_key_iv: "7208137db4a722e0cbcf6f8b467ef330"
server_key_version: "encryptionkey@0"
I20260620 20:19:14.791286 32179 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986536687152-17252-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986536687152-17252-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986536687152-17252-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:19:14.812542 32179 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:19:14.816829 32179 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:19:14.817113 32179 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:19:14.836047 32179 rpc_server.cc:307] RPC server started. Bound to: 127.16.217.62:36477
I20260620 20:19:14.836140 32245 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.217.62:36477 every 8 connection(s)
I20260620 20:19:14.837545 32179 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986536687152-17252-0/minicluster-data/master-0/data/info.pb
I20260620 20:19:14.844223 17252 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu as pid 32179
I20260620 20:19:14.845012 32246 sys_catalog.cc:263] Verifying existing consensus state
I20260620 20:19:14.845399 17252 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
/tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986536687152-17252-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986536687152-17252-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986536687152-17252-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986536687152-17252-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.217.1:44135
--local_ip_for_outbound_sockets=127.16.217.1
--tserver_master_addrs=127.16.217.62:36477
--webserver_port=38787
--webserver_interface=127.16.217.1
--keytab_file=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.1.keytab
--principal=kudu/127.16.217.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.217.20:38487
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986536687152-17252-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986536687152-17252-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986536687152-17252-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 20:19:14.850482 32246 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P becbda298fcf43ab9c61810d35e9764a: Bootstrap starting.
I20260620 20:19:14.873150 32246 log.cc:824] T 00000000000000000000000000000000 P becbda298fcf43ab9c61810d35e9764a: Log is configured to *not* fsync() on all Append() calls
I20260620 20:19:14.890661 32246 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P becbda298fcf43ab9c61810d35e9764a: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 20:19:14.891263 32246 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P becbda298fcf43ab9c61810d35e9764a: Bootstrap complete.
I20260620 20:19:14.901435 32246 raft_consensus.cc:348] T 00000000000000000000000000000000 P becbda298fcf43ab9c61810d35e9764a [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "becbda298fcf43ab9c61810d35e9764a" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 36477 } }
I20260620 20:19:14.902526 32246 raft_consensus.cc:729] T 00000000000000000000000000000000 P becbda298fcf43ab9c61810d35e9764a [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: becbda298fcf43ab9c61810d35e9764a, State: Initialized, Role: FOLLOWER
I20260620 20:19:14.903281 32246 consensus_queue.cc:260] T 00000000000000000000000000000000 P becbda298fcf43ab9c61810d35e9764a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "becbda298fcf43ab9c61810d35e9764a" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 36477 } }
I20260620 20:19:14.903529 32246 raft_consensus.cc:388] T 00000000000000000000000000000000 P becbda298fcf43ab9c61810d35e9764a [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 20:19:14.903666 32246 raft_consensus.cc:482] T 00000000000000000000000000000000 P becbda298fcf43ab9c61810d35e9764a [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 20:19:14.903864 32246 raft_consensus.cc:3037] T 00000000000000000000000000000000 P becbda298fcf43ab9c61810d35e9764a [term 2 FOLLOWER]: Advancing to term 3
I20260620 20:19:14.908367 32246 raft_consensus.cc:504] T 00000000000000000000000000000000 P becbda298fcf43ab9c61810d35e9764a [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "becbda298fcf43ab9c61810d35e9764a" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 36477 } }
I20260620 20:19:14.908911 32246 leader_election.cc:302] T 00000000000000000000000000000000 P becbda298fcf43ab9c61810d35e9764a [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: becbda298fcf43ab9c61810d35e9764a; no voters:
I20260620 20:19:14.909636 32246 leader_election.cc:288] T 00000000000000000000000000000000 P becbda298fcf43ab9c61810d35e9764a [CANDIDATE]: Term 3 election: Requested vote from peers
I20260620 20:19:14.909783 32250 raft_consensus.cc:2781] T 00000000000000000000000000000000 P becbda298fcf43ab9c61810d35e9764a [term 3 FOLLOWER]: Leader election won for term 3
I20260620 20:19:14.911070 32250 raft_consensus.cc:686] T 00000000000000000000000000000000 P becbda298fcf43ab9c61810d35e9764a [term 3 LEADER]: Becoming Leader. State: Replica: becbda298fcf43ab9c61810d35e9764a, State: Running, Role: LEADER
I20260620 20:19:14.911638 32250 consensus_queue.cc:237] T 00000000000000000000000000000000 P becbda298fcf43ab9c61810d35e9764a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "becbda298fcf43ab9c61810d35e9764a" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 36477 } }
I20260620 20:19:14.912006 32246 sys_catalog.cc:564] T 00000000000000000000000000000000 P becbda298fcf43ab9c61810d35e9764a [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:19:14.914574 32251 sys_catalog.cc:455] T 00000000000000000000000000000000 P becbda298fcf43ab9c61810d35e9764a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "becbda298fcf43ab9c61810d35e9764a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "becbda298fcf43ab9c61810d35e9764a" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 36477 } } }
I20260620 20:19:14.915050 32251 sys_catalog.cc:458] T 00000000000000000000000000000000 P becbda298fcf43ab9c61810d35e9764a [sys.catalog]: This master's current role is: LEADER
I20260620 20:19:14.917392 32252 sys_catalog.cc:455] T 00000000000000000000000000000000 P becbda298fcf43ab9c61810d35e9764a [sys.catalog]: SysCatalogTable state changed. Reason: New leader becbda298fcf43ab9c61810d35e9764a. Latest consensus state: current_term: 3 leader_uuid: "becbda298fcf43ab9c61810d35e9764a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "becbda298fcf43ab9c61810d35e9764a" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 36477 } } }
I20260620 20:19:14.917788 32252 sys_catalog.cc:458] T 00000000000000000000000000000000 P becbda298fcf43ab9c61810d35e9764a [sys.catalog]: This master's current role is: LEADER
I20260620 20:19:14.919308 32256 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 20:19:14.928337 32256 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 20:19:14.929888 32256 catalog_manager.cc:1259] Loaded cluster ID: a30997bc2e0d4a0b86613e6b28572b3a
I20260620 20:19:14.929991 32256 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260620 20:19:14.932137 32256 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7efeccfd59ad google::LogMessage::Fail() at ??:0
@ 0x7efeccfd888a google::LogMessage::SendToLog() at ??:0
@ 0x7efeccfd54de google::LogMessage::Flush() at ??:0
@ 0x7efeccfd6a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7efee2a0c512 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7efee2a09976 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7efee2a91610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7efee2a91439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7efecfc21345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7efecfc2491c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7efecfc24799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7efecfbffeaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7efed2ed96db start_thread at ??:0
@ 0x7efecbe6f61f clone at ??:0
W20260620 20:19:15.236128 32248 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:19:15.236390 32248 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:19:15.236546 32248 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:19:15.243750 32248 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:19:15.243850 32248 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:19:15.244046 32248 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.217.1
I20260620 20:19:15.253396 32248 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.217.20:38487
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986536687152-17252-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986536687152-17252-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.1.keytab
--principal=kudu/127.16.217.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.217.1:44135
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986536687152-17252-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.217.1
--webserver_port=38787
--tserver_master_addrs=127.16.217.62:36477
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.217.1
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986536687152-17252-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:19:15.254650 32248 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:19:15.255960 32248 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:19:15.269124 32287 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:19:15.269153 32288 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:19:15.269217 32290 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:19:15.270066 32248 server_base.cc:1034] running on GCE node
I20260620 20:19:15.270754 32248 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:19:15.271795 32248 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:19:15.273083 32248 hybrid_clock.cc:648] HybridClock initialized: now 1781986755273019 us; error 41 us; skew 500 ppm
I20260620 20:19:15.273443 32248 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:19:15 dist-test-slave-2hpg krb5kdc[31098](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986755, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.217.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:19:15.276870 32248 init.cc:377] Logged in from keytab as kudu/127.16.217.1@KRBTEST.COM (short username kudu)
I20260620 20:19:15.278368 32248 webserver.cc:460] Webserver started at http://127.16.217.1:38787/ using document root <none> and password file <none>
I20260620 20:19:15.279031 32248 fs_manager.cc:359] Metadata directory not provided
I20260620 20:19:15.279153 32248 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:19:15.284634 32248 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 20:19:15.287627 32297 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:19:15.288969 32248 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 20:19:15.289146 32248 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986536687152-17252-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986536687152-17252-0/minicluster-data/ts-0/wal
uuid: "8400677523bb4227b8e2d7b83e7f45ba"
format_stamp: "Formatted at 2026-06-20 20:19:09 on dist-test-slave-2hpg"
server_key: "1282b3fbdc54571751d10a364a758a0f"
server_key_iv: "b0843de0908b3105cefbbeef1b335a8a"
server_key_version: "encryptionkey@0"
I20260620 20:19:15.290287 32248 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986536687152-17252-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986536687152-17252-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986536687152-17252-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:19:15.303261 32248 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:19:15.307561 32248 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:19:15.307842 32248 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:19:15.308815 32248 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:19:15.310695 32304 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
W20260620 20:19:15.316368 32304 ts_tablet_manager.cc:1852] T 8b4678944eb04ee2856f77c1278ba65a P 8400677523bb4227b8e2d7b83e7f45ba: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260620 20:19:15.317399 32304 ts_tablet_manager.cc:1905] T 8b4678944eb04ee2856f77c1278ba65a P 8400677523bb4227b8e2d7b83e7f45ba: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:19:15.325026 32304 ts_tablet_manager.cc:1918] T 8b4678944eb04ee2856f77c1278ba65a P 8400677523bb4227b8e2d7b83e7f45ba: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 20:19:15.325332 32304 ts_tablet_manager.cc:1939] T 8b4678944eb04ee2856f77c1278ba65a P 8400677523bb4227b8e2d7b83e7f45ba: Deleting consensus metadata
I20260620 20:19:15.325765 32248 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260620 20:19:15.325915 32248 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.016s user 0.001s sys 0.000s
I20260620 20:19:15.326067 32248 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:19:15.326162 32248 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:19:15.360472 32248 rpc_server.cc:307] RPC server started. Bound to: 127.16.217.1:44135
I20260620 20:19:15.360530 32410 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.217.1:44135 every 8 connection(s)
I20260620 20:19:15.361815 32248 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986536687152-17252-0/minicluster-data/ts-0/data/info.pb
I20260620 20:19:15.366832 32413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:19:15.364973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:36477 (local address 127.16.217.1:49499)
0620 20:19:15.365712 (+ 739us) negotiation.cc:107] Waiting for socket to connect
0620 20:19:15.365988 (+ 276us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.217.62:36477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":480,"thread_start_us":292,"threads_started":1}
W20260620 20:19:15.367465 32411 heartbeater.cc:644] Failed to heartbeat to 127.16.217.62:36477 (0 consecutive failures): Network error: Failed to ping master at 127.16.217.62:36477: Client connection negotiation failed: client connection to 127.16.217.62:36477: connect: Connection refused (error 111)
I20260620 20:19:15.368372 32413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:19:15.367936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:36477 (local address 127.16.217.1:35773)
0620 20:19:15.368127 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0620 20:19:15.368218 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.217.62:36477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260620 20:19:15.369163 32413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:19:15.368853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:36477 (local address 127.16.217.1:43145)
0620 20:19:15.368964 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0620 20:19:15.369035 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.217.62:36477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260620 20:19:15.369385 32411 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 20:19:15.371521 17252 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu as pid 32248
I20260620 20:19:15.372238 17252 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
/tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986536687152-17252-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986536687152-17252-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986536687152-17252-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986536687152-17252-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.217.2:33683
--local_ip_for_outbound_sockets=127.16.217.2
--tserver_master_addrs=127.16.217.62:36477
--webserver_port=44817
--webserver_interface=127.16.217.2
--keytab_file=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.2.keytab
--principal=kudu/127.16.217.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.217.20:38487
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986536687152-17252-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986536687152-17252-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986536687152-17252-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:19:15.715271 32414 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:19:15.715518 32414 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:19:15.715665 32414 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:19:15.723356 32414 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:19:15.723453 32414 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:19:15.723623 32414 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.217.2
I20260620 20:19:15.733156 32414 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.217.20:38487
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986536687152-17252-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986536687152-17252-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.2.keytab
--principal=kudu/127.16.217.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.217.2:33683
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986536687152-17252-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.217.2
--webserver_port=44817
--tserver_master_addrs=127.16.217.62:36477
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.217.2
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986536687152-17252-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:19:15.734429 32414 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:19:15.735690 32414 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:19:15.749190 32419 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:19:15.749120 32420 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:19:15.749230 32422 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:19:15.750516 32414 server_base.cc:1034] running on GCE node
I20260620 20:19:15.751113 32414 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:19:15.752038 32414 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:19:15.753255 32414 hybrid_clock.cc:648] HybridClock initialized: now 1781986755753201 us; error 45 us; skew 500 ppm
I20260620 20:19:15.753568 32414 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:19:15 dist-test-slave-2hpg krb5kdc[31098](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986755, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.217.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:19:15.756589 32414 init.cc:377] Logged in from keytab as kudu/127.16.217.2@KRBTEST.COM (short username kudu)
I20260620 20:19:15.758046 32414 webserver.cc:460] Webserver started at http://127.16.217.2:44817/ using document root <none> and password file <none>
I20260620 20:19:15.758718 32414 fs_manager.cc:359] Metadata directory not provided
I20260620 20:19:15.758857 32414 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:19:15.764079 32414 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 20:19:15.766680 32429 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:19:15.767992 32414 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 20:19:15.768162 32414 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986536687152-17252-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986536687152-17252-0/minicluster-data/ts-1/wal
uuid: "89eac4104e6c4573b766d41f6999395e"
format_stamp: "Formatted at 2026-06-20 20:19:10 on dist-test-slave-2hpg"
server_key: "322fa0b1cbcc51e6e4d913d5dde29c77"
server_key_iv: "12d9c585b627ebd71485d82d57331537"
server_key_version: "encryptionkey@0"
I20260620 20:19:15.769160 32414 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986536687152-17252-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986536687152-17252-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986536687152-17252-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:19:15.782859 32414 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:19:15.786855 32414 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:19:15.787115 32414 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:19:15.788033 32414 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:19:15.789354 32414 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:19:15.789481 32414 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:19:15.789603 32414 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:19:15.789688 32414 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:19:15.823359 32414 rpc_server.cc:307] RPC server started. Bound to: 127.16.217.2:33683
I20260620 20:19:15.823415 32541 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.217.2:33683 every 8 connection(s)
I20260620 20:19:15.824846 32414 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986536687152-17252-0/minicluster-data/ts-1/data/info.pb
I20260620 20:19:15.829505 32544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:19:15.827740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:36477 (local address 127.16.217.2:59553)
0620 20:19:15.828446 (+ 706us) negotiation.cc:107] Waiting for socket to connect
0620 20:19:15.828651 (+ 205us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.217.62:36477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":445,"thread_start_us":264,"threads_started":1}
W20260620 20:19:15.830255 32542 heartbeater.cc:644] Failed to heartbeat to 127.16.217.62:36477 (0 consecutive failures): Network error: Failed to ping master at 127.16.217.62:36477: Client connection negotiation failed: client connection to 127.16.217.62:36477: connect: Connection refused (error 111)
I20260620 20:19:15.831043 32544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:19:15.830753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:36477 (local address 127.16.217.2:40395)
0620 20:19:15.830879 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0620 20:19:15.830943 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.217.62:36477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260620 20:19:15.831982 32544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:19:15.831581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:36477 (local address 127.16.217.2:33327)
0620 20:19:15.831767 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0620 20:19:15.831859 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.217.62:36477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
W20260620 20:19:15.832252 32542 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 20:19:15.834204 17252 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu as pid 32414
I20260620 20:19:15.834816 17252 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
/tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986536687152-17252-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986536687152-17252-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986536687152-17252-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986536687152-17252-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.217.3:34191
--local_ip_for_outbound_sockets=127.16.217.3
--tserver_master_addrs=127.16.217.62:36477
--webserver_port=39173
--webserver_interface=127.16.217.3
--keytab_file=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.3.keytab
--principal=kudu/127.16.217.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.217.20:38487
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986536687152-17252-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986536687152-17252-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986536687152-17252-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:19:16.154840 32545 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:19:16.155164 32545 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:19:16.155318 32545 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:19:16.162464 32545 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:19:16.162609 32545 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:19:16.162783 32545 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.217.3
I20260620 20:19:16.172533 32545 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.217.20:38487
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986536687152-17252-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986536687152-17252-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.3.keytab
--principal=kudu/127.16.217.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.217.3:34191
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986536687152-17252-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.217.3
--webserver_port=39173
--tserver_master_addrs=127.16.217.62:36477
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.217.3
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986536687152-17252-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:19:16.174003 32545 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:19:16.175354 32545 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:19:16.188596 32553 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:19:16.188680 32551 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:19:16.188675 32550 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:19:16.189898 32545 server_base.cc:1034] running on GCE node
I20260620 20:19:16.190555 32545 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:19:16.191526 32545 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:19:16.192746 32545 hybrid_clock.cc:648] HybridClock initialized: now 1781986756192699 us; error 44 us; skew 500 ppm
I20260620 20:19:16.193059 32545 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:19:16 dist-test-slave-2hpg krb5kdc[31098](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986756, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.217.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:19:16.196342 32545 init.cc:377] Logged in from keytab as kudu/127.16.217.3@KRBTEST.COM (short username kudu)
I20260620 20:19:16.197942 32545 webserver.cc:460] Webserver started at http://127.16.217.3:39173/ using document root <none> and password file <none>
I20260620 20:19:16.198624 32545 fs_manager.cc:359] Metadata directory not provided
I20260620 20:19:16.198761 32545 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:19:16.203868 32545 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.005s
I20260620 20:19:16.206609 32560 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:19:16.207844 32545 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260620 20:19:16.208009 32545 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986536687152-17252-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986536687152-17252-0/minicluster-data/ts-2/wal
uuid: "5ee69ada9a214bc8a73ad8e534401c60"
format_stamp: "Formatted at 2026-06-20 20:19:10 on dist-test-slave-2hpg"
server_key: "affbce870a7beb9fcad9c5c2b6384c61"
server_key_iv: "9b939c1a3d3d41a24e1720b1879e1358"
server_key_version: "encryptionkey@0"
I20260620 20:19:16.208963 32545 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986536687152-17252-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986536687152-17252-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986536687152-17252-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:19:16.224267 32545 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:19:16.228122 32545 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:19:16.228379 32545 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:19:16.229323 32545 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:19:16.230654 32545 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:19:16.230770 32545 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:19:16.230898 32545 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:19:16.230983 32545 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:19:16.263376 32545 rpc_server.cc:307] RPC server started. Bound to: 127.16.217.3:34191
I20260620 20:19:16.263460 32672 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.217.3:34191 every 8 connection(s)
I20260620 20:19:16.264864 32545 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986536687152-17252-0/minicluster-data/ts-2/data/info.pb
I20260620 20:19:16.266024 17252 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu as pid 32545
I20260620 20:19:16.270332 32676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:19:16.268485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:36477 (local address 127.16.217.3:48775)
0620 20:19:16.269187 (+ 702us) negotiation.cc:107] Waiting for socket to connect
0620 20:19:16.269384 (+ 197us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.217.62:36477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":275,"threads_started":1}
W20260620 20:19:16.271034 32673 heartbeater.cc:644] Failed to heartbeat to 127.16.217.62:36477 (0 consecutive failures): Network error: Failed to ping master at 127.16.217.62:36477: Client connection negotiation failed: client connection to 127.16.217.62:36477: connect: Connection refused (error 111)
I20260620 20:19:16.271823 32676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:19:16.271491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:36477 (local address 127.16.217.3:37621)
0620 20:19:16.271653 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0620 20:19:16.271721 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.217.62:36477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260620 20:19:16.272789 32676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:19:16.272387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:36477 (local address 127.16.217.3:39137)
0620 20:19:16.272569 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0620 20:19:16.272677 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.217.62:36477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
W20260620 20:19:16.273068 32673 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 20:19:16.370658 32677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:19:16.369917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:36477 (local address 127.16.217.1:38181)
0620 20:19:16.370366 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0620 20:19:16.370443 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.217.62:36477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":349,"thread_start_us":220,"threads_started":1}
I20260620 20:19:16.833591 32678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:19:16.832799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:36477 (local address 127.16.217.2:41275)
0620 20:19:16.833255 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0620 20:19:16.833373 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.217.62:36477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":350,"thread_start_us":225,"threads_started":1}
I20260620 20:19:17.274492 32679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:19:17.273542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:36477 (local address 127.16.217.3:55301)
0620 20:19:17.274105 (+ 563us) negotiation.cc:107] Waiting for socket to connect
0620 20:19:17.274226 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.217.62:36477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":454,"thread_start_us":321,"threads_started":1}
I20260620 20:19:17.372381 32680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:19:17.371606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:36477 (local address 127.16.217.1:50227)
0620 20:19:17.372100 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0620 20:19:17.372189 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.217.62:36477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":392,"thread_start_us":225,"threads_started":1}
I20260620 20:19:17.835189 32681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:19:17.834500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:36477 (local address 127.16.217.2:34061)
0620 20:19:17.834939 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0620 20:19:17.835013 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.217.62:36477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":338,"thread_start_us":201,"threads_started":1}
I20260620 20:19:18.276075 32682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:19:18.275379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:36477 (local address 127.16.217.3:53083)
0620 20:19:18.275831 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0620 20:19:18.275910 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.217.62:36477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":236,"threads_started":1}
I20260620 20:19:18.374125 32683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:19:18.373270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:36477 (local address 127.16.217.1:50329)
0620 20:19:18.373820 (+ 550us) negotiation.cc:107] Waiting for socket to connect
0620 20:19:18.373937 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.217.62:36477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":303,"threads_started":1}
I20260620 20:19:18.836764 32684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:19:18.836040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:36477 (local address 127.16.217.2:56993)
0620 20:19:18.836486 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0620 20:19:18.836563 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.217.62:36477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":343,"thread_start_us":239,"threads_started":1}
I20260620 20:19:19.277685 32685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:19:19.276915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:36477 (local address 127.16.217.3:33223)
0620 20:19:19.277361 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0620 20:19:19.277439 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.217.62:36477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":351,"thread_start_us":237,"threads_started":1}
I20260620 20:19:19.375684 32686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:19:19.374958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:36477 (local address 127.16.217.1:35599)
0620 20:19:19.375430 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0620 20:19:19.375509 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.217.62:36477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":233,"threads_started":1}
I20260620 20:19:19.838627 32687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:19:19.837741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:36477 (local address 127.16.217.2:42317)
0620 20:19:19.838366 (+ 625us) negotiation.cc:107] Waiting for socket to connect
0620 20:19:19.838443 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.217.62:36477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":538,"thread_start_us":344,"threads_started":1}
I20260620 20:19:20.279284 32688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:19:20.278549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:36477 (local address 127.16.217.3:46949)
0620 20:19:20.278998 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0620 20:19:20.279087 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.217.62:36477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":350,"thread_start_us":225,"threads_started":1}
I20260620 20:19:20.377223 32689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:19:20.376502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:36477 (local address 127.16.217.1:57419)
0620 20:19:20.376956 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0620 20:19:20.377045 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.217.62:36477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":343,"thread_start_us":230,"threads_started":1}
I20260620 20:19:20.840220 32690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:19:20.839530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:36477 (local address 127.16.217.2:49975)
0620 20:19:20.839956 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0620 20:19:20.840037 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.217.62:36477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":229,"threads_started":1}
I20260620 20:19:21.280931 32691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:19:21.280166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:36477 (local address 127.16.217.3:51713)
0620 20:19:21.280668 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0620 20:19:21.280762 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.217.62:36477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":390,"thread_start_us":245,"threads_started":1}
I20260620 20:19:21.378898 32692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:19:21.378231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:36477 (local address 127.16.217.1:45781)
0620 20:19:21.378644 (+ 413us) negotiation.cc:107] Waiting for socket to connect
0620 20:19:21.378727 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.217.62:36477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":315,"thread_start_us":206,"threads_started":1}
I20260620 20:19:21.841943 32693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:19:21.841140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:36477 (local address 127.16.217.2:45581)
0620 20:19:21.841621 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0620 20:19:21.841724 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.217.62:36477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":372,"thread_start_us":258,"threads_started":1}
I20260620 20:19:22.282671 32694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:19:22.281958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:36477 (local address 127.16.217.3:42109)
0620 20:19:22.282420 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0620 20:19:22.282512 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.217.62:36477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":243,"threads_started":1}
I20260620 20:19:22.380780 32695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:19:22.379893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:36477 (local address 127.16.217.1:44217)
0620 20:19:22.380463 (+ 570us) negotiation.cc:107] Waiting for socket to connect
0620 20:19:22.380554 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.217.62:36477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":458,"thread_start_us":285,"threads_started":1}
I20260620 20:19:22.843695 32696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:19:22.842849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:36477 (local address 127.16.217.2:49243)
0620 20:19:22.843359 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0620 20:19:22.843460 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.217.62:36477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":231,"threads_started":1}
I20260620 20:19:23.284433 32697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:19:23.283628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:36477 (local address 127.16.217.3:37963)
0620 20:19:23.284134 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0620 20:19:23.284222 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.217.62:36477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":219,"threads_started":1}
I20260620 20:19:23.382718 32698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:19:23.381826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:36477 (local address 127.16.217.1:56089)
0620 20:19:23.382457 (+ 631us) negotiation.cc:107] Waiting for socket to connect
0620 20:19:23.382541 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.217.62:36477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":541,"thread_start_us":308,"threads_started":1}
I20260620 20:19:23.845750 32699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:19:23.844931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:36477 (local address 127.16.217.2:43937)
0620 20:19:23.845437 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0620 20:19:23.845546 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.217.62:36477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":257,"threads_started":1}
I20260620 20:19:24.286161 32700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:19:24.285392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:36477 (local address 127.16.217.3:43403)
0620 20:19:24.285840 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0620 20:19:24.285983 (+ 143us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.217.62:36477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":238,"threads_started":1}
I20260620 20:19:24.384545 32701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:19:24.383678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:36477 (local address 127.16.217.1:46841)
0620 20:19:24.384298 (+ 620us) negotiation.cc:107] Waiting for socket to connect
0620 20:19:24.384389 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.217.62:36477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":477,"thread_start_us":273,"threads_started":1}
I20260620 20:19:24.847442 32702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:19:24.846723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:36477 (local address 127.16.217.2:52795)
0620 20:19:24.847186 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0620 20:19:24.847265 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.217.62:36477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":368,"thread_start_us":236,"threads_started":1}
I20260620 20:19:25.287765 32703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:19:25.287029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:36477 (local address 127.16.217.3:49835)
0620 20:19:25.287498 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0620 20:19:25.287580 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.217.62:36477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":233,"threads_started":1}
I20260620 20:19:25.386340 32704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:19:25.385561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:36477 (local address 127.16.217.1:57867)
0620 20:19:25.386072 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0620 20:19:25.386163 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.217.62:36477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":412,"thread_start_us":245,"threads_started":1}
I20260620 20:19:25.849241 32705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:19:25.848399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:36477 (local address 127.16.217.2:53151)
0620 20:19:25.848875 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0620 20:19:25.849003 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.217.62:36477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":359,"thread_start_us":226,"threads_started":1}
I20260620 20:19:26.289536 32706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:19:26.288831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:36477 (local address 127.16.217.3:50631)
0620 20:19:26.289289 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0620 20:19:26.289377 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.217.62:36477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":229,"threads_started":1}
I20260620 20:19:26.388024 32707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:19:26.387259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:36477 (local address 127.16.217.1:50779)
0620 20:19:26.387758 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0620 20:19:26.387848 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.217.62:36477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":388,"thread_start_us":271,"threads_started":1}
I20260620 20:19:26.851169 32708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:19:26.850315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:36477 (local address 127.16.217.2:53851)
0620 20:19:26.850801 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0620 20:19:26.850892 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.217.62:36477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":242,"threads_started":1}
I20260620 20:19:27.291313 32709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:19:27.290591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:36477 (local address 127.16.217.3:48069)
0620 20:19:27.291038 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0620 20:19:27.291138 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.217.62:36477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":349,"thread_start_us":226,"threads_started":1}
I20260620 20:19:27.389657 32710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:19:27.388963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:36477 (local address 127.16.217.1:36001)
0620 20:19:27.389408 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0620 20:19:27.389484 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.217.62:36477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":258,"threads_started":1}
I20260620 20:19:27.852922 32711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:19:27.852222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:36477 (local address 127.16.217.2:36117)
0620 20:19:27.852690 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0620 20:19:27.852771 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.217.62:36477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":257,"threads_started":1}
I20260620 20:19:28.292900 32712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:19:28.292237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:36477 (local address 127.16.217.3:60731)
0620 20:19:28.292664 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0620 20:19:28.292748 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.217.62:36477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":313,"thread_start_us":209,"threads_started":1}
I20260620 20:19:28.391559 32713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:19:28.390708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:36477 (local address 127.16.217.1:34421)
0620 20:19:28.391256 (+ 548us) negotiation.cc:107] Waiting for socket to connect
0620 20:19:28.391372 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.217.62:36477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":445,"thread_start_us":275,"threads_started":1}
I20260620 20:19:28.854629 32714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:19:28.853885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:36477 (local address 127.16.217.2:34037)
0620 20:19:28.854337 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0620 20:19:28.854441 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.217.62:36477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":234,"threads_started":1}
I20260620 20:19:29.294744 32715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:19:29.293912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:36477 (local address 127.16.217.3:41113)
0620 20:19:29.294401 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0620 20:19:29.294550 (+ 149us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.217.62:36477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":382,"thread_start_us":244,"threads_started":1}
I20260620 20:19:29.393281 32716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:19:29.392478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:36477 (local address 127.16.217.1:42583)
0620 20:19:29.392955 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0620 20:19:29.393060 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.217.62:36477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":361,"thread_start_us":243,"threads_started":1}
I20260620 20:19:29.856120 32717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:19:29.855434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:36477 (local address 127.16.217.2:57133)
0620 20:19:29.855851 (+ 417us) negotiation.cc:107] Waiting for socket to connect
0620 20:19:29.855928 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.217.62:36477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":325,"thread_start_us":220,"threads_started":1}
I20260620 20:19:30.296511 32718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:19:30.295713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:36477 (local address 127.16.217.3:42737)
0620 20:19:30.296179 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0620 20:19:30.296264 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.217.62:36477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":356,"thread_start_us":231,"threads_started":1}
I20260620 20:19:30.395412 32719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:19:30.394659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:36477 (local address 127.16.217.1:37023)
0620 20:19:30.395155 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0620 20:19:30.395243 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.217.62:36477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":379,"thread_start_us":258,"threads_started":1}
I20260620 20:19:30.858075 32720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:19:30.857269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:36477 (local address 127.16.217.2:42887)
0620 20:19:30.857746 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0620 20:19:30.857836 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.217.62:36477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":338,"thread_start_us":226,"threads_started":1}
I20260620 20:19:31.277307 17252 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu pid 32248
I20260620 20:19:31.298975 32721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:19:31.278635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59792 (local address 127.16.217.1:44135)
0620 20:19:31.279314 (+ 679us) server_negotiation.cc:206] Beginning negotiation
0620 20:19:31.279340 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0620 20:19:31.279411 (+ 71us) server_negotiation.cc:407] Connection header received
0620 20:19:31.281757 (+ 2346us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:19:31.281771 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:19:31.282371 (+ 600us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:19:31.282709 (+ 338us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:19:31.284416 (+ 1707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:19:31.285725 (+ 1309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:19:31.287263 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:19:31.288006 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:19:31.288209 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:19:31.291451 (+ 3242us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:19:31.291461 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:19:31.291478 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:19:31.291523 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:19:31.295091 (+ 3568us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:19:31.295885 (+ 794us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:19:31.295893 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:19:31.295901 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:19:31.296113 (+ 212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:19:31.296529 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:19:31.296534 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:19:31.296536 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:19:31.297273 (+ 737us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:19:31.297588 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0620 20:19:31.298194 (+ 606us) server_negotiation.cc:299] Negotiation successful
0620 20:19:31.298540 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":475,"thread_start_us":336,"threads_started":1}
I20260620 20:19:31.299437 32722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:19:31.297893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:36477 (local address 127.16.217.3:50457)
0620 20:19:31.299093 (+ 1200us) negotiation.cc:107] Waiting for socket to connect
0620 20:19:31.299226 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.217.62:36477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1071,"thread_start_us":961,"threads_started":1}
I20260620 20:19:31.300602 32406 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:19:31.386586 17252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu with pid 32248
I20260620 20:19:31.406795 17252 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu pid 32414
I20260620 20:19:31.424894 32724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:19:31.407851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57262 (local address 127.16.217.2:33683)
0620 20:19:31.408431 (+ 580us) server_negotiation.cc:206] Beginning negotiation
0620 20:19:31.408446 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 20:19:31.408481 (+ 35us) server_negotiation.cc:407] Connection header received
0620 20:19:31.409976 (+ 1495us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:19:31.409986 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:19:31.410417 (+ 431us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:19:31.410767 (+ 350us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:19:31.412226 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:19:31.413530 (+ 1304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:19:31.414884 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:19:31.415566 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:19:31.415783 (+ 217us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:19:31.418584 (+ 2801us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:19:31.418592 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:19:31.418606 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:19:31.418647 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:19:31.421539 (+ 2892us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:19:31.422301 (+ 762us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:19:31.422309 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:19:31.422316 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:19:31.422475 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:19:31.422871 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:19:31.422875 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:19:31.422877 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:19:31.423435 (+ 558us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:19:31.423767 (+ 332us) server_negotiation.cc:1035] Waiting for connection context
0620 20:19:31.424206 (+ 439us) server_negotiation.cc:299] Negotiation successful
0620 20:19:31.424612 (+ 406us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":370,"thread_start_us":254,"threads_started":1}
I20260620 20:19:31.426462 32537 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:19:31.511081 17252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu with pid 32414
I20260620 20:19:31.529116 17252 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu pid 32545
I20260620 20:19:31.547137 32726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:19:31.530157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40274 (local address 127.16.217.3:34191)
0620 20:19:31.530799 (+ 642us) server_negotiation.cc:206] Beginning negotiation
0620 20:19:31.530813 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 20:19:31.530856 (+ 43us) server_negotiation.cc:407] Connection header received
0620 20:19:31.532473 (+ 1617us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:19:31.532481 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:19:31.532937 (+ 456us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:19:31.533228 (+ 291us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:19:31.534842 (+ 1614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:19:31.536091 (+ 1249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:19:31.537358 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:19:31.538204 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:19:31.538357 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:19:31.540917 (+ 2560us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:19:31.540924 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:19:31.540940 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:19:31.540977 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:19:31.543990 (+ 3013us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:19:31.544647 (+ 657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:19:31.544654 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:19:31.544661 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:19:31.544825 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:19:31.545173 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:19:31.545176 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:19:31.545178 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:19:31.545722 (+ 544us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:19:31.546019 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0620 20:19:31.546374 (+ 355us) server_negotiation.cc:299] Negotiation successful
0620 20:19:31.546835 (+ 461us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":489,"thread_start_us":321,"threads_started":1}
I20260620 20:19:31.548518 32668 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:19:31.634222 17252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu with pid 32545
2026-06-20T20:19:31Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (24483 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986536687152-17252-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 20:19:31 dist-test-slave-2hpg krb5kdc[32730](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 20:19:31 dist-test-slave-2hpg krb5kdc[32730](info): set up 2 sockets
Jun 20 20:19:31 dist-test-slave-2hpg krb5kdc[32730](info): commencing operation
krb5kdc: starting...
W20260620 20:19:32.822729 17252 mini_kdc.cc:121] Time spent starting KDC: real 1.149s user 0.002s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986536687152-17252-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986536687152-17252-0/krb5kdc/test-user.keytab.
Jun 20 20:19:32 dist-test-slave-2hpg krb5kdc[32730](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986772, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T20:19:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T20:19:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.217.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.217.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.217.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.62.keytab.
Entry for principal kudu/127.16.217.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.62.keytab.
I20260620 20:19:33.055557 17252 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
/tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986536687152-17252-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986536687152-17252-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986536687152-17252-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986536687152-17252-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.217.62:34019
--webserver_interface=127.16.217.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.62.keytab
--principal=kudu/127.16.217.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.217.20:35087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.217.62:34019
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986536687152-17252-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986536687152-17252-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986536687152-17252-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:19:33.387080 32743 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:19:33.387398 32743 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:19:33.387491 32743 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:19:33.394761 32743 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260620 20:19:33.394863 32743 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260620 20:19:33.394903 32743 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260620 20:19:33.394948 32743 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:19:33.395007 32743 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:19:33.395076 32743 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:19:33.395120 32743 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:19:33.395183 32743 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:19:33.404963 32743 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.217.20:35087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986536687152-17252-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986536687152-17252-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.16.217.62:34019
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.62.keytab
--principal=kudu/127.16.217.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.217.62:34019
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986536687152-17252-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.217.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986536687152-17252-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:19:33.406292 32743 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:19:33.407591 32743 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:19:33.419762 32751 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:19:33.419771 32748 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:19:33.419826 32749 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:19:33.420439 32743 server_base.cc:1034] running on GCE node
I20260620 20:19:33.421197 32743 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:19:33.422699 32743 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:19:33.423920 32743 hybrid_clock.cc:648] HybridClock initialized: now 1781986773423864 us; error 24 us; skew 500 ppm
I20260620 20:19:33.424273 32743 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:19:33 dist-test-slave-2hpg krb5kdc[32730](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986773, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.217.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:19:33.427611 32743 init.cc:377] Logged in from keytab as kudu/127.16.217.62@KRBTEST.COM (short username kudu)
I20260620 20:19:33.429090 32743 webserver.cc:460] Webserver started at http://127.16.217.62:35859/ using document root <none> and password file <none>
I20260620 20:19:33.429745 32743 fs_manager.cc:359] Metadata directory not provided
I20260620 20:19:33.429831 32743 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:19:33.430161 32743 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:19:33.433162 32743 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986536687152-17252-0/minicluster-data/master-0/data/instance:
uuid: "8e39d1abbd4e46a7bc73a63bfb83dfd0"
format_stamp: "Formatted at 2026-06-20 20:19:33 on dist-test-slave-2hpg"
server_key: "e31576e9a31de7cbfbaa3f0936591892"
server_key_iv: "1cea50ed55f4321d6496cb1ca04baf58"
server_key_version: "encryptionkey@0"
I20260620 20:19:33.433940 32743 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986536687152-17252-0/minicluster-data/master-0/wal/instance:
uuid: "8e39d1abbd4e46a7bc73a63bfb83dfd0"
format_stamp: "Formatted at 2026-06-20 20:19:33 on dist-test-slave-2hpg"
server_key: "e31576e9a31de7cbfbaa3f0936591892"
server_key_iv: "1cea50ed55f4321d6496cb1ca04baf58"
server_key_version: "encryptionkey@0"
I20260620 20:19:33.438886 32743 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 20:19:33.442203 32758 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:19:33.443517 32743 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 20:19:33.443792 32743 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986536687152-17252-0/minicluster-data/master-0/data,/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986536687152-17252-0/minicluster-data/master-0/wal
uuid: "8e39d1abbd4e46a7bc73a63bfb83dfd0"
format_stamp: "Formatted at 2026-06-20 20:19:33 on dist-test-slave-2hpg"
server_key: "e31576e9a31de7cbfbaa3f0936591892"
server_key_iv: "1cea50ed55f4321d6496cb1ca04baf58"
server_key_version: "encryptionkey@0"
I20260620 20:19:33.444029 32743 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986536687152-17252-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986536687152-17252-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986536687152-17252-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:19:33.468060 32743 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:19:33.472793 32743 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:19:33.473090 32743 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:19:33.492539 32743 rpc_server.cc:307] RPC server started. Bound to: 127.16.217.62:34019
I20260620 20:19:33.492565 341 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.217.62:34019 every 8 connection(s)
I20260620 20:19:33.494021 32743 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986536687152-17252-0/minicluster-data/master-0/data/info.pb
I20260620 20:19:33.497822 17252 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu as pid 32743
I20260620 20:19:33.498042 342 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:19:33.498059 17252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986536687152-17252-0/minicluster-data/master-0/wal/instance
I20260620 20:19:33.498926 17252 external_mini_cluster.cc:1428] Setting key c93f5cc38937cde1d18015231c7332b8
I20260620 20:19:33.513566 342 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8e39d1abbd4e46a7bc73a63bfb83dfd0: Bootstrap starting.
Jun 20 20:19:33 dist-test-slave-2hpg krb5kdc[32730](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986772, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.217.62@KRBTEST.COM
I20260620 20:19:33.516917 342 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8e39d1abbd4e46a7bc73a63bfb83dfd0: Neither blocks nor log segments found. Creating new log.
I20260620 20:19:33.518095 342 log.cc:824] T 00000000000000000000000000000000 P 8e39d1abbd4e46a7bc73a63bfb83dfd0: Log is configured to *not* fsync() on all Append() calls
I20260620 20:19:33.520923 342 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8e39d1abbd4e46a7bc73a63bfb83dfd0: No bootstrap required, opened a new log
I20260620 20:19:33.522616 345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:19:33.500954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43418 (local address 127.16.217.62:34019)
0620 20:19:33.501766 (+ 812us) server_negotiation.cc:206] Beginning negotiation
0620 20:19:33.501777 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 20:19:33.501816 (+ 39us) server_negotiation.cc:407] Connection header received
0620 20:19:33.503349 (+ 1533us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:19:33.503370 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:19:33.503740 (+ 370us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:19:33.504065 (+ 325us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:19:33.505961 (+ 1896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:19:33.507216 (+ 1255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:19:33.509296 (+ 2080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:19:33.510139 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:19:33.510296 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:19:33.515281 (+ 4985us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:19:33.515290 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:19:33.515303 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:19:33.515345 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:19:33.518519 (+ 3174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:19:33.519261 (+ 742us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:19:33.519268 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:19:33.519275 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:19:33.519444 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:19:33.519836 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:19:33.519842 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:19:33.519845 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:19:33.520354 (+ 509us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:19:33.520669 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0620 20:19:33.521094 (+ 425us) server_negotiation.cc:299] Negotiation successful
0620 20:19:33.521559 (+ 465us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":609,"thread_start_us":394,"threads_started":1}
I20260620 20:19:33.529300 342 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8e39d1abbd4e46a7bc73a63bfb83dfd0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e39d1abbd4e46a7bc73a63bfb83dfd0" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 34019 } }
I20260620 20:19:33.529681 342 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8e39d1abbd4e46a7bc73a63bfb83dfd0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:19:33.529795 342 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8e39d1abbd4e46a7bc73a63bfb83dfd0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e39d1abbd4e46a7bc73a63bfb83dfd0, State: Initialized, Role: FOLLOWER
I20260620 20:19:33.530367 342 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8e39d1abbd4e46a7bc73a63bfb83dfd0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e39d1abbd4e46a7bc73a63bfb83dfd0" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 34019 } }
I20260620 20:19:33.530627 342 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8e39d1abbd4e46a7bc73a63bfb83dfd0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 20:19:33.530769 342 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8e39d1abbd4e46a7bc73a63bfb83dfd0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 20:19:33.530937 342 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8e39d1abbd4e46a7bc73a63bfb83dfd0 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:19:33.533106 342 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8e39d1abbd4e46a7bc73a63bfb83dfd0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e39d1abbd4e46a7bc73a63bfb83dfd0" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 34019 } }
I20260620 20:19:33.533584 342 leader_election.cc:302] T 00000000000000000000000000000000 P 8e39d1abbd4e46a7bc73a63bfb83dfd0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8e39d1abbd4e46a7bc73a63bfb83dfd0; no voters:
I20260620 20:19:33.534261 342 leader_election.cc:288] T 00000000000000000000000000000000 P 8e39d1abbd4e46a7bc73a63bfb83dfd0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 20:19:33.534369 347 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8e39d1abbd4e46a7bc73a63bfb83dfd0 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:19:33.535140 347 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8e39d1abbd4e46a7bc73a63bfb83dfd0 [term 1 LEADER]: Becoming Leader. State: Replica: 8e39d1abbd4e46a7bc73a63bfb83dfd0, State: Running, Role: LEADER
I20260620 20:19:33.535698 347 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8e39d1abbd4e46a7bc73a63bfb83dfd0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e39d1abbd4e46a7bc73a63bfb83dfd0" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 34019 } }
I20260620 20:19:33.536547 342 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8e39d1abbd4e46a7bc73a63bfb83dfd0 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:19:33.538733 349 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8e39d1abbd4e46a7bc73a63bfb83dfd0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8e39d1abbd4e46a7bc73a63bfb83dfd0. Latest consensus state: current_term: 1 leader_uuid: "8e39d1abbd4e46a7bc73a63bfb83dfd0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e39d1abbd4e46a7bc73a63bfb83dfd0" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 34019 } } }
I20260620 20:19:33.539253 349 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8e39d1abbd4e46a7bc73a63bfb83dfd0 [sys.catalog]: This master's current role is: LEADER
I20260620 20:19:33.539218 348 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8e39d1abbd4e46a7bc73a63bfb83dfd0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8e39d1abbd4e46a7bc73a63bfb83dfd0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e39d1abbd4e46a7bc73a63bfb83dfd0" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 34019 } } }
I20260620 20:19:33.540174 348 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8e39d1abbd4e46a7bc73a63bfb83dfd0 [sys.catalog]: This master's current role is: LEADER
I20260620 20:19:33.542455 354 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 20:19:33.548732 354 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 20:19:33.558059 354 catalog_manager.cc:1347] Generated new cluster ID: e0dabc224e8647618b3b459dc9e53e35
I20260620 20:19:33.558193 354 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 20:19:33.578907 354 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 20:19:33.580821 354 catalog_manager.cc:1504] Loading token signing keys...
I20260620 20:19:33.591219 354 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8e39d1abbd4e46a7bc73a63bfb83dfd0: Generated new TSK 0
I20260620 20:19:33.592460 354 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.217.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.217.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.217.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.1.keytab.
Entry for principal kudu/127.16.217.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.1.keytab.
I20260620 20:19:33.654582 17252 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
/tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986536687152-17252-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986536687152-17252-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986536687152-17252-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986536687152-17252-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.217.1:0
--local_ip_for_outbound_sockets=127.16.217.1
--webserver_interface=127.16.217.1
--webserver_port=0
--tserver_master_addrs=127.16.217.62:34019
--keytab_file=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.1.keytab
--principal=kudu/127.16.217.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.217.20:35087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986536687152-17252-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986536687152-17252-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986536687152-17252-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:19:33.984977 368 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:19:33.985234 368 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:19:33.985406 368 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:19:33.993201 368 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:19:33.993299 368 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:19:33.993476 368 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.217.1
I20260620 20:19:34.003609 368 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.217.20:35087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986536687152-17252-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986536687152-17252-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.1.keytab
--principal=kudu/127.16.217.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.217.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986536687152-17252-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.217.1
--webserver_port=0
--tserver_master_addrs=127.16.217.62:34019
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.217.1
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986536687152-17252-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:19:34.004922 368 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:19:34.006394 368 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:19:34.020165 374 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:19:34.020201 373 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:19:34.020401 376 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:19:34.021036 368 server_base.cc:1034] running on GCE node
I20260620 20:19:34.021660 368 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:19:34.022776 368 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:19:34.024053 368 hybrid_clock.cc:648] HybridClock initialized: now 1781986774023976 us; error 93 us; skew 500 ppm
I20260620 20:19:34.024384 368 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:19:34 dist-test-slave-2hpg krb5kdc[32730](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986774, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.217.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:19:34.027554 368 init.cc:377] Logged in from keytab as kudu/127.16.217.1@KRBTEST.COM (short username kudu)
I20260620 20:19:34.028991 368 webserver.cc:460] Webserver started at http://127.16.217.1:46685/ using document root <none> and password file <none>
I20260620 20:19:34.029670 368 fs_manager.cc:359] Metadata directory not provided
I20260620 20:19:34.029810 368 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:19:34.030133 368 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:19:34.032960 368 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986536687152-17252-0/minicluster-data/ts-0/data/instance:
uuid: "48ddf12e1ab742819abba58564f9d6b5"
format_stamp: "Formatted at 2026-06-20 20:19:34 on dist-test-slave-2hpg"
server_key: "a59ce49777766b69fa009b135592f350"
server_key_iv: "f28fb0e2561f6d706f9cf30682c90572"
server_key_version: "encryptionkey@0"
I20260620 20:19:34.033702 368 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986536687152-17252-0/minicluster-data/ts-0/wal/instance:
uuid: "48ddf12e1ab742819abba58564f9d6b5"
format_stamp: "Formatted at 2026-06-20 20:19:34 on dist-test-slave-2hpg"
server_key: "a59ce49777766b69fa009b135592f350"
server_key_iv: "f28fb0e2561f6d706f9cf30682c90572"
server_key_version: "encryptionkey@0"
I20260620 20:19:34.038957 368 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 20:19:34.042313 383 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:19:34.043581 368 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260620 20:19:34.043747 368 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986536687152-17252-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986536687152-17252-0/minicluster-data/ts-0/wal
uuid: "48ddf12e1ab742819abba58564f9d6b5"
format_stamp: "Formatted at 2026-06-20 20:19:34 on dist-test-slave-2hpg"
server_key: "a59ce49777766b69fa009b135592f350"
server_key_iv: "f28fb0e2561f6d706f9cf30682c90572"
server_key_version: "encryptionkey@0"
I20260620 20:19:34.043932 368 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986536687152-17252-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986536687152-17252-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986536687152-17252-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:19:34.067729 368 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:19:34.071717 368 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:19:34.072021 368 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:19:34.072957 368 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:19:34.074306 368 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:19:34.074414 368 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:19:34.074556 368 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:19:34.074628 368 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:19:34.108417 368 rpc_server.cc:307] RPC server started. Bound to: 127.16.217.1:38751
I20260620 20:19:34.108465 495 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.217.1:38751 every 8 connection(s)
I20260620 20:19:34.109756 368 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986536687152-17252-0/minicluster-data/ts-0/data/info.pb
I20260620 20:19:34.117338 17252 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu as pid 368
I20260620 20:19:34.117553 17252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986536687152-17252-0/minicluster-data/ts-0/wal/instance
I20260620 20:19:34.118242 17252 external_mini_cluster.cc:1428] Setting key 8fb6cebd5d5c4143d02ab1397fb8d97a
Jun 20 20:19:34 dist-test-slave-2hpg krb5kdc[32730](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986774, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.217.1@KRBTEST.COM for kudu/127.16.217.62@KRBTEST.COM
I20260620 20:19:34.129694 498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:19:34.112472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.217.1:49429 (local address 127.16.217.62:34019)
0620 20:19:34.112993 (+ 521us) server_negotiation.cc:206] Beginning negotiation
0620 20:19:34.112997 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:19:34.113843 (+ 846us) server_negotiation.cc:407] Connection header received
0620 20:19:34.115700 (+ 1857us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:19:34.115708 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:19:34.115799 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:19:34.115915 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:19:34.117337 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:19:34.118266 (+ 929us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:19:34.119909 (+ 1643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:19:34.120309 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:19:34.120439 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:19:34.124071 (+ 3632us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:19:34.124076 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:19:34.124079 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:19:34.124111 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:19:34.126746 (+ 2635us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:19:34.127458 (+ 712us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:19:34.127466 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:19:34.127470 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:19:34.127591 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:19:34.128003 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:19:34.128009 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:19:34.128012 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:19:34.128272 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:19:34.128424 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0620 20:19:34.129091 (+ 667us) server_negotiation.cc:299] Negotiation successful
0620 20:19:34.129319 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":397,"thread_start_us":302,"threads_started":1}
I20260620 20:19:34.130637 499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:19:34.112923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:34019 (local address 127.16.217.1:49429)
0620 20:19:34.113692 (+ 769us) negotiation.cc:107] Waiting for socket to connect
0620 20:19:34.113738 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 20:19:34.115421 (+ 1683us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:19:34.116151 (+ 730us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:19:34.116159 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:19:34.116648 (+ 489us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:19:34.117174 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:19:34.117190 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:19:34.118417 (+ 1227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:19:34.118422 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:19:34.119766 (+ 1344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:19:34.119778 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:19:34.120532 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:19:34.120542 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:19:34.120764 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:19:34.121457 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:19:34.121477 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:19:34.123859 (+ 2382us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:19:34.126889 (+ 3030us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:19:34.126897 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:19:34.126913 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:19:34.127335 (+ 422us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:19:34.127709 (+ 374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:19:34.127713 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:19:34.127717 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:19:34.127892 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:19:34.128398 (+ 506us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:19:34.128406 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:19:34.128729 (+ 323us) client_negotiation.cc:769] Sending connection context
0620 20:19:34.128988 (+ 259us) client_negotiation.cc:240] Negotiation successful
0620 20:19:34.129508 (+ 520us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":515,"thread_start_us":262,"threads_started":1}
I20260620 20:19:34.132222 496 heartbeater.cc:347] Connected to a master server at 127.16.217.62:34019
I20260620 20:19:34.132552 496 heartbeater.cc:464] Registering TS with master...
I20260620 20:19:34.133241 496 heartbeater.cc:511] Master 127.16.217.62:34019 requested a full tablet report, sending...
I20260620 20:19:34.135672 307 ts_manager.cc:194] Registered new tserver with Master: 48ddf12e1ab742819abba58564f9d6b5 (127.16.217.1:38751)
I20260620 20:19:34.137949 307 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.217.1@KRBTEST.COM'} at 127.16.217.1:49429
WARNING: no policy specified for kudu/127.16.217.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.217.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.217.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.2.keytab.
Entry for principal kudu/127.16.217.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.2.keytab.
I20260620 20:19:34.171525 17252 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
/tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986536687152-17252-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986536687152-17252-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986536687152-17252-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986536687152-17252-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.217.2:0
--local_ip_for_outbound_sockets=127.16.217.2
--webserver_interface=127.16.217.2
--webserver_port=0
--tserver_master_addrs=127.16.217.62:34019
--keytab_file=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.2.keytab
--principal=kudu/127.16.217.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.217.20:35087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986536687152-17252-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986536687152-17252-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986536687152-17252-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:19:34.496341 502 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:19:34.496611 502 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:19:34.496773 502 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:19:34.504122 502 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:19:34.504222 502 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:19:34.504400 502 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.217.2
I20260620 20:19:34.514243 502 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.217.20:35087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986536687152-17252-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986536687152-17252-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.2.keytab
--principal=kudu/127.16.217.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.217.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986536687152-17252-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.217.2
--webserver_port=0
--tserver_master_addrs=127.16.217.62:34019
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.217.2
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986536687152-17252-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:19:34.515537 502 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:19:34.516840 502 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:19:34.529944 507 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:19:34.529928 510 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:19:34.530061 508 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:19:34.530758 502 server_base.cc:1034] running on GCE node
I20260620 20:19:34.531688 502 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:19:34.532675 502 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:19:34.533949 502 hybrid_clock.cc:648] HybridClock initialized: now 1781986774533913 us; error 56 us; skew 500 ppm
I20260620 20:19:34.534373 502 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:19:34 dist-test-slave-2hpg krb5kdc[32730](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986774, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.217.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:19:34.537794 502 init.cc:377] Logged in from keytab as kudu/127.16.217.2@KRBTEST.COM (short username kudu)
I20260620 20:19:34.539278 502 webserver.cc:460] Webserver started at http://127.16.217.2:44433/ using document root <none> and password file <none>
I20260620 20:19:34.539934 502 fs_manager.cc:359] Metadata directory not provided
I20260620 20:19:34.540040 502 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:19:34.540313 502 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:19:34.543359 502 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986536687152-17252-0/minicluster-data/ts-1/data/instance:
uuid: "bb9261ea26ae46a580415794f87a5ee0"
format_stamp: "Formatted at 2026-06-20 20:19:34 on dist-test-slave-2hpg"
server_key: "7b9d4c4e8c73a23e07f972a525e02b22"
server_key_iv: "caa256f0c4f6f0763ef305f7f306aecd"
server_key_version: "encryptionkey@0"
I20260620 20:19:34.544132 502 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986536687152-17252-0/minicluster-data/ts-1/wal/instance:
uuid: "bb9261ea26ae46a580415794f87a5ee0"
format_stamp: "Formatted at 2026-06-20 20:19:34 on dist-test-slave-2hpg"
server_key: "7b9d4c4e8c73a23e07f972a525e02b22"
server_key_iv: "caa256f0c4f6f0763ef305f7f306aecd"
server_key_version: "encryptionkey@0"
I20260620 20:19:34.550043 502 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260620 20:19:34.553474 517 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:19:34.554874 502 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 20:19:34.555053 502 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986536687152-17252-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986536687152-17252-0/minicluster-data/ts-1/wal
uuid: "bb9261ea26ae46a580415794f87a5ee0"
format_stamp: "Formatted at 2026-06-20 20:19:34 on dist-test-slave-2hpg"
server_key: "7b9d4c4e8c73a23e07f972a525e02b22"
server_key_iv: "caa256f0c4f6f0763ef305f7f306aecd"
server_key_version: "encryptionkey@0"
I20260620 20:19:34.555248 502 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986536687152-17252-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986536687152-17252-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986536687152-17252-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:19:34.580399 502 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:19:34.584702 502 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:19:34.585002 502 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:19:34.586053 502 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:19:34.587381 502 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:19:34.587455 502 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:19:34.587638 502 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:19:34.587694 502 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:19:34.620049 502 rpc_server.cc:307] RPC server started. Bound to: 127.16.217.2:33863
I20260620 20:19:34.620098 629 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.217.2:33863 every 8 connection(s)
I20260620 20:19:34.621351 502 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986536687152-17252-0/minicluster-data/ts-1/data/info.pb
I20260620 20:19:34.624739 17252 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu as pid 502
I20260620 20:19:34.624974 17252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986536687152-17252-0/minicluster-data/ts-1/wal/instance
I20260620 20:19:34.625716 17252 external_mini_cluster.cc:1428] Setting key 51b76664a65988142dd3588f0fca0108
Jun 20 20:19:34 dist-test-slave-2hpg krb5kdc[32730](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986774, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.217.2@KRBTEST.COM for kudu/127.16.217.62@KRBTEST.COM
I20260620 20:19:34.645267 498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:19:34.623965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.217.2:58879 (local address 127.16.217.62:34019)
0620 20:19:34.624197 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0620 20:19:34.624204 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:19:34.625475 (+ 1271us) server_negotiation.cc:407] Connection header received
0620 20:19:34.627525 (+ 2050us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:19:34.627531 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:19:34.627646 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:19:34.627772 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:19:34.629822 (+ 2050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:19:34.631110 (+ 1288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:19:34.633480 (+ 2370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:19:34.634246 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:19:34.634387 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:19:34.638146 (+ 3759us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:19:34.638156 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:19:34.638164 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:19:34.638211 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:19:34.641811 (+ 3600us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:19:34.642633 (+ 822us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:19:34.642641 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:19:34.642648 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:19:34.642798 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:19:34.643241 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:19:34.643250 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:19:34.643256 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:19:34.643632 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:19:34.643794 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0620 20:19:34.644560 (+ 766us) server_negotiation.cc:299] Negotiation successful
0620 20:19:34.644810 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260620 20:19:34.646381 632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:19:34.624448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:34019 (local address 127.16.217.2:58879)
0620 20:19:34.625283 (+ 835us) negotiation.cc:107] Waiting for socket to connect
0620 20:19:34.625333 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0620 20:19:34.627260 (+ 1927us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:19:34.628015 (+ 755us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:19:34.628026 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:19:34.628756 (+ 730us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:19:34.629572 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:19:34.629601 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:19:34.631332 (+ 1731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:19:34.631341 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:19:34.633318 (+ 1977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:19:34.633333 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:19:34.634471 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:19:34.634477 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:19:34.634708 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:19:34.635430 (+ 722us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:19:34.635455 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:19:34.637849 (+ 2394us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:19:34.642005 (+ 4156us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:19:34.642012 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:19:34.642024 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:19:34.642503 (+ 479us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:19:34.642917 (+ 414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:19:34.642920 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:19:34.642922 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:19:34.643129 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:19:34.643803 (+ 674us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:19:34.643810 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:19:34.644117 (+ 307us) client_negotiation.cc:769] Sending connection context
0620 20:19:34.644435 (+ 318us) client_negotiation.cc:240] Negotiation successful
0620 20:19:34.644964 (+ 529us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":538,"thread_start_us":348,"threads_started":1}
I20260620 20:19:34.647933 630 heartbeater.cc:347] Connected to a master server at 127.16.217.62:34019
I20260620 20:19:34.648245 630 heartbeater.cc:464] Registering TS with master...
I20260620 20:19:34.649094 630 heartbeater.cc:511] Master 127.16.217.62:34019 requested a full tablet report, sending...
I20260620 20:19:34.650616 307 ts_manager.cc:194] Registered new tserver with Master: bb9261ea26ae46a580415794f87a5ee0 (127.16.217.2:33863)
I20260620 20:19:34.651430 307 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.217.2@KRBTEST.COM'} at 127.16.217.2:58879
WARNING: no policy specified for kudu/127.16.217.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.217.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.217.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.3.keytab.
Entry for principal kudu/127.16.217.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.3.keytab.
I20260620 20:19:34.680914 17252 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
/tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986536687152-17252-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986536687152-17252-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986536687152-17252-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986536687152-17252-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.217.3:0
--local_ip_for_outbound_sockets=127.16.217.3
--webserver_interface=127.16.217.3
--webserver_port=0
--tserver_master_addrs=127.16.217.62:34019
--keytab_file=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.3.keytab
--principal=kudu/127.16.217.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.217.20:35087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986536687152-17252-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986536687152-17252-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986536687152-17252-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:19:35.000036 635 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:19:35.000386 635 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:19:35.000567 635 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:19:35.007795 635 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:19:35.007920 635 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:19:35.008111 635 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.217.3
I20260620 20:19:35.018258 635 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.217.20:35087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986536687152-17252-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986536687152-17252-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.3.keytab
--principal=kudu/127.16.217.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.217.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986536687152-17252-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.217.3
--webserver_port=0
--tserver_master_addrs=127.16.217.62:34019
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.217.3
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986536687152-17252-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:19:35.019533 635 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:19:35.020967 635 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:19:35.034179 640 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:19:35.034200 641 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:19:35.034235 643 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:19:35.035182 635 server_base.cc:1034] running on GCE node
I20260620 20:19:35.035822 635 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:19:35.036794 635 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:19:35.037978 635 hybrid_clock.cc:648] HybridClock initialized: now 1781986775037941 us; error 31 us; skew 500 ppm
I20260620 20:19:35.038290 635 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:19:35 dist-test-slave-2hpg krb5kdc[32730](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986775, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.217.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:19:35.041395 635 init.cc:377] Logged in from keytab as kudu/127.16.217.3@KRBTEST.COM (short username kudu)
I20260620 20:19:35.042773 635 webserver.cc:460] Webserver started at http://127.16.217.3:38299/ using document root <none> and password file <none>
I20260620 20:19:35.043388 635 fs_manager.cc:359] Metadata directory not provided
I20260620 20:19:35.043483 635 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:19:35.043726 635 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:19:35.046658 635 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986536687152-17252-0/minicluster-data/ts-2/data/instance:
uuid: "24811d2b3eec4754a9194eabaa73b41f"
format_stamp: "Formatted at 2026-06-20 20:19:35 on dist-test-slave-2hpg"
server_key: "4d2eb5a7b0d6dc44451589d8f4e00b44"
server_key_iv: "cf7eb6dbad246eff8cf648f969b186d1"
server_key_version: "encryptionkey@0"
I20260620 20:19:35.047335 635 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986536687152-17252-0/minicluster-data/ts-2/wal/instance:
uuid: "24811d2b3eec4754a9194eabaa73b41f"
format_stamp: "Formatted at 2026-06-20 20:19:35 on dist-test-slave-2hpg"
server_key: "4d2eb5a7b0d6dc44451589d8f4e00b44"
server_key_iv: "cf7eb6dbad246eff8cf648f969b186d1"
server_key_version: "encryptionkey@0"
I20260620 20:19:35.052369 635 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 20:19:35.055649 650 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:19:35.056845 635 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 20:19:35.057003 635 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986536687152-17252-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986536687152-17252-0/minicluster-data/ts-2/wal
uuid: "24811d2b3eec4754a9194eabaa73b41f"
format_stamp: "Formatted at 2026-06-20 20:19:35 on dist-test-slave-2hpg"
server_key: "4d2eb5a7b0d6dc44451589d8f4e00b44"
server_key_iv: "cf7eb6dbad246eff8cf648f969b186d1"
server_key_version: "encryptionkey@0"
I20260620 20:19:35.057206 635 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986536687152-17252-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986536687152-17252-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986536687152-17252-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:19:35.075807 635 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:19:35.080005 635 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:19:35.080322 635 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:19:35.081251 635 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:19:35.082564 635 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:19:35.082639 635 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:19:35.082768 635 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:19:35.082818 635 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:19:35.118583 635 rpc_server.cc:307] RPC server started. Bound to: 127.16.217.3:46007
I20260620 20:19:35.118635 762 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.217.3:46007 every 8 connection(s)
I20260620 20:19:35.119850 635 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986536687152-17252-0/minicluster-data/ts-2/data/info.pb
I20260620 20:19:35.122371 17252 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu as pid 635
I20260620 20:19:35.122561 17252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986536687152-17252-0/minicluster-data/ts-2/wal/instance
I20260620 20:19:35.123021 17252 external_mini_cluster.cc:1428] Setting key 67049f8d9afcf66e6f3fa3f2deca216e
Jun 20 20:19:35 dist-test-slave-2hpg krb5kdc[32730](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986775, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.217.3@KRBTEST.COM for kudu/127.16.217.62@KRBTEST.COM
I20260620 20:19:35.139156 498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:19:35.122423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.217.3:47789 (local address 127.16.217.62:34019)
0620 20:19:35.122621 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0620 20:19:35.122627 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:19:35.123942 (+ 1315us) server_negotiation.cc:407] Connection header received
0620 20:19:35.125976 (+ 2034us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:19:35.125982 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:19:35.126107 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:19:35.126196 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:19:35.127582 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:19:35.128313 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:19:35.129778 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:19:35.130224 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:19:35.130317 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:19:35.133482 (+ 3165us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:19:35.133486 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:19:35.133489 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:19:35.133519 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:19:35.136034 (+ 2515us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:19:35.136821 (+ 787us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:19:35.136830 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:19:35.136841 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:19:35.136988 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:19:35.137418 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:19:35.137424 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:19:35.137427 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:19:35.137755 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:19:35.137929 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0620 20:19:35.138604 (+ 675us) server_negotiation.cc:299] Negotiation successful
0620 20:19:35.138822 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260620 20:19:35.140131 765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:19:35.122907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:34019 (local address 127.16.217.3:47789)
0620 20:19:35.123775 (+ 868us) negotiation.cc:107] Waiting for socket to connect
0620 20:19:35.123837 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0620 20:19:35.125653 (+ 1816us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:19:35.126410 (+ 757us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:19:35.126431 (+ 21us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:19:35.126905 (+ 474us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:19:35.127432 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:19:35.127449 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:19:35.128439 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:19:35.128443 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:19:35.129662 (+ 1219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:19:35.129674 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:19:35.130400 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:19:35.130405 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:19:35.130600 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:19:35.131252 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:19:35.131278 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:19:35.133319 (+ 2041us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:19:35.136147 (+ 2828us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:19:35.136153 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:19:35.136163 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:19:35.136708 (+ 545us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:19:35.137120 (+ 412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:19:35.137128 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:19:35.137132 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:19:35.137306 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:19:35.137904 (+ 598us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:19:35.137912 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:19:35.138229 (+ 317us) client_negotiation.cc:769] Sending connection context
0620 20:19:35.138484 (+ 255us) client_negotiation.cc:240] Negotiation successful
0620 20:19:35.138961 (+ 477us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":562,"thread_start_us":352,"threads_started":1}
I20260620 20:19:35.141053 496 heartbeater.cc:503] Master 127.16.217.62:34019 was elected leader, sending a full tablet report...
I20260620 20:19:35.141729 763 heartbeater.cc:347] Connected to a master server at 127.16.217.62:34019
I20260620 20:19:35.142053 763 heartbeater.cc:464] Registering TS with master...
I20260620 20:19:35.142899 763 heartbeater.cc:511] Master 127.16.217.62:34019 requested a full tablet report, sending...
I20260620 20:19:35.144243 307 ts_manager.cc:194] Registered new tserver with Master: 24811d2b3eec4754a9194eabaa73b41f (127.16.217.3:46007)
I20260620 20:19:35.145066 307 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.217.3@KRBTEST.COM'} at 127.16.217.3:47789
I20260620 20:19:35.150404 17252 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 20:19:35.168496 498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:19:35.154900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43424 (local address 127.16.217.62:34019)
0620 20:19:35.155056 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0620 20:19:35.155061 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:19:35.155513 (+ 452us) server_negotiation.cc:407] Connection header received
0620 20:19:35.155657 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:19:35.155661 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:19:35.155723 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:19:35.155801 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:19:35.157386 (+ 1585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:19:35.158234 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:19:35.159725 (+ 1491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:19:35.160156 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:19:35.160303 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:19:35.162814 (+ 2511us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:19:35.162819 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:19:35.162823 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:19:35.162857 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:19:35.166146 (+ 3289us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:19:35.166875 (+ 729us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:19:35.166880 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:19:35.166882 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:19:35.167016 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:19:35.167331 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:19:35.167334 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:19:35.167336 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:19:35.167690 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:19:35.167871 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0620 20:19:35.168115 (+ 244us) server_negotiation.cc:299] Negotiation successful
0620 20:19:35.168272 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 20:19:35.172329 307 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43424:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 20:19:35.179126 307 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 20:19:35.223065 778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:19:35.198316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47892 (local address 127.16.217.3:46007)
0620 20:19:35.200452 (+ 2136us) server_negotiation.cc:206] Beginning negotiation
0620 20:19:35.200467 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 20:19:35.211969 (+ 11502us) server_negotiation.cc:407] Connection header received
0620 20:19:35.213704 (+ 1735us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:19:35.213717 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:19:35.214090 (+ 373us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:19:35.214290 (+ 200us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:19:35.216020 (+ 1730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:19:35.217690 (+ 1670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:19:35.220595 (+ 2905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:19:35.221733 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:19:35.221927 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:19:35.222239 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0620 20:19:35.222509 (+ 270us) server_negotiation.cc:299] Negotiation successful
0620 20:19:35.222689 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1884,"thread_start_us":5230,"threads_started":1}
I20260620 20:19:35.223065 779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:19:35.191983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.3:46007 (local address 127.0.0.1:47892)
0620 20:19:35.211837 (+ 19854us) negotiation.cc:107] Waiting for socket to connect
0620 20:19:35.211857 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 20:19:35.213396 (+ 1539us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:19:35.214711 (+ 1315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:19:35.214719 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:19:35.214754 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:19:35.215739 (+ 985us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:19:35.215761 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:19:35.217842 (+ 2081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:19:35.217851 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:19:35.220424 (+ 2573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:19:35.220436 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:19:35.221996 (+ 1560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:19:35.222003 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:19:35.222224 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:19:35.222290 (+ 66us) client_negotiation.cc:769] Sending connection context
0620 20:19:35.222439 (+ 149us) client_negotiation.cc:240] Negotiation successful
0620 20:19:35.222660 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19723,"thread_start_us":23372,"threads_started":1}
I20260620 20:19:35.224721 775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:19:35.191512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.1:38751 (local address 127.0.0.1:32876)
0620 20:19:35.204298 (+ 12786us) negotiation.cc:107] Waiting for socket to connect
0620 20:19:35.204350 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0620 20:19:35.204538 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:19:35.216602 (+ 12064us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:19:35.216606 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:19:35.216629 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:19:35.217107 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:19:35.217134 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:19:35.219216 (+ 2082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:19:35.219222 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:19:35.221891 (+ 2669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:19:35.221905 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:19:35.223776 (+ 1871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:19:35.223783 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:19:35.223961 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:19:35.223997 (+ 36us) client_negotiation.cc:769] Sending connection context
0620 20:19:35.224110 (+ 113us) client_negotiation.cc:240] Negotiation successful
0620 20:19:35.224251 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12524,"mutex_wait_us":111,"thread_start_us":12393,"threads_started":1}
I20260620 20:19:35.225725 777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:19:35.205299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.2:33863 (local address 127.0.0.1:53338)
0620 20:19:35.214614 (+ 9315us) negotiation.cc:107] Waiting for socket to connect
0620 20:19:35.214649 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 20:19:35.214811 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:19:35.216955 (+ 2144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:19:35.216961 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:19:35.216989 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:19:35.217409 (+ 420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:19:35.217424 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:19:35.219535 (+ 2111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:19:35.219556 (+ 21us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:19:35.221902 (+ 2346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:19:35.221915 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:19:35.224918 (+ 3003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:19:35.224924 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:19:35.225102 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:19:35.225147 (+ 45us) client_negotiation.cc:769] Sending connection context
0620 20:19:35.225253 (+ 106us) client_negotiation.cc:240] Negotiation successful
0620 20:19:35.225392 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9145,"mutex_wait_us":34,"thread_start_us":6679,"threads_started":1}
I20260620 20:19:35.225795 780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:19:35.205320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53338 (local address 127.16.217.2:33863)
0620 20:19:35.214167 (+ 8847us) server_negotiation.cc:206] Beginning negotiation
0620 20:19:35.214177 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 20:19:35.214996 (+ 819us) server_negotiation.cc:407] Connection header received
0620 20:19:35.215123 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:19:35.215131 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:19:35.215393 (+ 262us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:19:35.217602 (+ 2209us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:19:35.217868 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:19:35.219353 (+ 1485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:19:35.222083 (+ 2730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:19:35.223538 (+ 1455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:19:35.223684 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:19:35.223868 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0620 20:19:35.225290 (+ 1422us) server_negotiation.cc:299] Negotiation successful
0620 20:19:35.225458 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8632,"thread_start_us":8471,"threads_started":1}
I20260620 20:19:35.226902 776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:19:35.191791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32876 (local address 127.16.217.1:38751)
0620 20:19:35.216062 (+ 24271us) server_negotiation.cc:206] Beginning negotiation
0620 20:19:35.216069 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:19:35.216090 (+ 21us) server_negotiation.cc:407] Connection header received
0620 20:19:35.216159 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:19:35.216163 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:19:35.216409 (+ 246us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:19:35.216618 (+ 209us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:19:35.217358 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:19:35.219043 (+ 1685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:19:35.222083 (+ 3040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:19:35.223538 (+ 1455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:19:35.223682 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:19:35.223901 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0620 20:19:35.224152 (+ 251us) server_negotiation.cc:299] Negotiation successful
0620 20:19:35.224325 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24049,"thread_start_us":23873,"threads_started":1}
I20260620 20:19:35.227543 698 tablet_service.cc:1467] Processing CreateTablet for tablet adcc815954884850873c99f877af9be8 (DEFAULT_TABLE table=test-table [id=a5b5c8c4f0d646888a59c37ea3a0a6dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:19:35.230077 698 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet adcc815954884850873c99f877af9be8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:19:35.229517 565 tablet_service.cc:1467] Processing CreateTablet for tablet adcc815954884850873c99f877af9be8 (DEFAULT_TABLE table=test-table [id=a5b5c8c4f0d646888a59c37ea3a0a6dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:19:35.230194 431 tablet_service.cc:1467] Processing CreateTablet for tablet adcc815954884850873c99f877af9be8 (DEFAULT_TABLE table=test-table [id=a5b5c8c4f0d646888a59c37ea3a0a6dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:19:35.231055 565 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet adcc815954884850873c99f877af9be8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:19:35.231303 431 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet adcc815954884850873c99f877af9be8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:19:35.243191 781 tablet_bootstrap.cc:492] T adcc815954884850873c99f877af9be8 P 48ddf12e1ab742819abba58564f9d6b5: Bootstrap starting.
I20260620 20:19:35.246194 782 tablet_bootstrap.cc:492] T adcc815954884850873c99f877af9be8 P 24811d2b3eec4754a9194eabaa73b41f: Bootstrap starting.
I20260620 20:19:35.246618 781 tablet_bootstrap.cc:654] T adcc815954884850873c99f877af9be8 P 48ddf12e1ab742819abba58564f9d6b5: Neither blocks nor log segments found. Creating new log.
I20260620 20:19:35.248041 781 log.cc:824] T adcc815954884850873c99f877af9be8 P 48ddf12e1ab742819abba58564f9d6b5: Log is configured to *not* fsync() on all Append() calls
I20260620 20:19:35.248133 783 tablet_bootstrap.cc:492] T adcc815954884850873c99f877af9be8 P bb9261ea26ae46a580415794f87a5ee0: Bootstrap starting.
I20260620 20:19:35.250344 782 tablet_bootstrap.cc:654] T adcc815954884850873c99f877af9be8 P 24811d2b3eec4754a9194eabaa73b41f: Neither blocks nor log segments found. Creating new log.
I20260620 20:19:35.251772 783 tablet_bootstrap.cc:654] T adcc815954884850873c99f877af9be8 P bb9261ea26ae46a580415794f87a5ee0: Neither blocks nor log segments found. Creating new log.
I20260620 20:19:35.251740 782 log.cc:824] T adcc815954884850873c99f877af9be8 P 24811d2b3eec4754a9194eabaa73b41f: Log is configured to *not* fsync() on all Append() calls
I20260620 20:19:35.253116 783 log.cc:824] T adcc815954884850873c99f877af9be8 P bb9261ea26ae46a580415794f87a5ee0: Log is configured to *not* fsync() on all Append() calls
I20260620 20:19:35.253204 781 tablet_bootstrap.cc:492] T adcc815954884850873c99f877af9be8 P 48ddf12e1ab742819abba58564f9d6b5: No bootstrap required, opened a new log
I20260620 20:19:35.253479 781 ts_tablet_manager.cc:1397] T adcc815954884850873c99f877af9be8 P 48ddf12e1ab742819abba58564f9d6b5: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.003s
I20260620 20:19:35.255730 782 tablet_bootstrap.cc:492] T adcc815954884850873c99f877af9be8 P 24811d2b3eec4754a9194eabaa73b41f: No bootstrap required, opened a new log
I20260620 20:19:35.255987 782 ts_tablet_manager.cc:1397] T adcc815954884850873c99f877af9be8 P 24811d2b3eec4754a9194eabaa73b41f: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260620 20:19:35.257053 783 tablet_bootstrap.cc:492] T adcc815954884850873c99f877af9be8 P bb9261ea26ae46a580415794f87a5ee0: No bootstrap required, opened a new log
I20260620 20:19:35.257311 783 ts_tablet_manager.cc:1397] T adcc815954884850873c99f877af9be8 P bb9261ea26ae46a580415794f87a5ee0: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.006s
I20260620 20:19:35.264575 781 raft_consensus.cc:348] T adcc815954884850873c99f877af9be8 P 48ddf12e1ab742819abba58564f9d6b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48ddf12e1ab742819abba58564f9d6b5" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 38751 } } peers { permanent_uuid: "24811d2b3eec4754a9194eabaa73b41f" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 46007 } } peers { permanent_uuid: "bb9261ea26ae46a580415794f87a5ee0" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 33863 } }
I20260620 20:19:35.265048 781 raft_consensus.cc:374] T adcc815954884850873c99f877af9be8 P 48ddf12e1ab742819abba58564f9d6b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:19:35.265199 781 raft_consensus.cc:729] T adcc815954884850873c99f877af9be8 P 48ddf12e1ab742819abba58564f9d6b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48ddf12e1ab742819abba58564f9d6b5, State: Initialized, Role: FOLLOWER
I20260620 20:19:35.265847 781 consensus_queue.cc:260] T adcc815954884850873c99f877af9be8 P 48ddf12e1ab742819abba58564f9d6b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48ddf12e1ab742819abba58564f9d6b5" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 38751 } } peers { permanent_uuid: "24811d2b3eec4754a9194eabaa73b41f" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 46007 } } peers { permanent_uuid: "bb9261ea26ae46a580415794f87a5ee0" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 33863 } }
I20260620 20:19:35.266160 783 raft_consensus.cc:348] T adcc815954884850873c99f877af9be8 P bb9261ea26ae46a580415794f87a5ee0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48ddf12e1ab742819abba58564f9d6b5" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 38751 } } peers { permanent_uuid: "24811d2b3eec4754a9194eabaa73b41f" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 46007 } } peers { permanent_uuid: "bb9261ea26ae46a580415794f87a5ee0" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 33863 } }
I20260620 20:19:35.266702 783 raft_consensus.cc:374] T adcc815954884850873c99f877af9be8 P bb9261ea26ae46a580415794f87a5ee0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:19:35.266849 783 raft_consensus.cc:729] T adcc815954884850873c99f877af9be8 P bb9261ea26ae46a580415794f87a5ee0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb9261ea26ae46a580415794f87a5ee0, State: Initialized, Role: FOLLOWER
I20260620 20:19:35.267449 783 consensus_queue.cc:260] T adcc815954884850873c99f877af9be8 P bb9261ea26ae46a580415794f87a5ee0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48ddf12e1ab742819abba58564f9d6b5" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 38751 } } peers { permanent_uuid: "24811d2b3eec4754a9194eabaa73b41f" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 46007 } } peers { permanent_uuid: "bb9261ea26ae46a580415794f87a5ee0" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 33863 } }
I20260620 20:19:35.267668 781 ts_tablet_manager.cc:1428] T adcc815954884850873c99f877af9be8 P 48ddf12e1ab742819abba58564f9d6b5: Time spent starting tablet: real 0.014s user 0.007s sys 0.007s
I20260620 20:19:35.266610 782 raft_consensus.cc:348] T adcc815954884850873c99f877af9be8 P 24811d2b3eec4754a9194eabaa73b41f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48ddf12e1ab742819abba58564f9d6b5" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 38751 } } peers { permanent_uuid: "24811d2b3eec4754a9194eabaa73b41f" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 46007 } } peers { permanent_uuid: "bb9261ea26ae46a580415794f87a5ee0" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 33863 } }
I20260620 20:19:35.268237 782 raft_consensus.cc:374] T adcc815954884850873c99f877af9be8 P 24811d2b3eec4754a9194eabaa73b41f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:19:35.268402 782 raft_consensus.cc:729] T adcc815954884850873c99f877af9be8 P 24811d2b3eec4754a9194eabaa73b41f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 24811d2b3eec4754a9194eabaa73b41f, State: Initialized, Role: FOLLOWER
I20260620 20:19:35.268610 630 heartbeater.cc:503] Master 127.16.217.62:34019 was elected leader, sending a full tablet report...
I20260620 20:19:35.269315 783 ts_tablet_manager.cc:1428] T adcc815954884850873c99f877af9be8 P bb9261ea26ae46a580415794f87a5ee0: Time spent starting tablet: real 0.012s user 0.011s sys 0.001s
I20260620 20:19:35.269170 782 consensus_queue.cc:260] T adcc815954884850873c99f877af9be8 P 24811d2b3eec4754a9194eabaa73b41f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48ddf12e1ab742819abba58564f9d6b5" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 38751 } } peers { permanent_uuid: "24811d2b3eec4754a9194eabaa73b41f" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 46007 } } peers { permanent_uuid: "bb9261ea26ae46a580415794f87a5ee0" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 33863 } }
I20260620 20:19:35.270563 763 heartbeater.cc:503] Master 127.16.217.62:34019 was elected leader, sending a full tablet report...
I20260620 20:19:35.272552 782 ts_tablet_manager.cc:1428] T adcc815954884850873c99f877af9be8 P 24811d2b3eec4754a9194eabaa73b41f: Time spent starting tablet: real 0.016s user 0.014s sys 0.000s
W20260620 20:19:35.362205 497 tablet.cc:2369] T adcc815954884850873c99f877af9be8 P 48ddf12e1ab742819abba58564f9d6b5: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 20:19:35.371419 764 tablet.cc:2369] T adcc815954884850873c99f877af9be8 P 24811d2b3eec4754a9194eabaa73b41f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 20:19:35.373345 631 tablet.cc:2369] T adcc815954884850873c99f877af9be8 P bb9261ea26ae46a580415794f87a5ee0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 20:19:35.552291 788 raft_consensus.cc:482] T adcc815954884850873c99f877af9be8 P bb9261ea26ae46a580415794f87a5ee0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 20:19:35.552655 788 raft_consensus.cc:504] T adcc815954884850873c99f877af9be8 P bb9261ea26ae46a580415794f87a5ee0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48ddf12e1ab742819abba58564f9d6b5" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 38751 } } peers { permanent_uuid: "24811d2b3eec4754a9194eabaa73b41f" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 46007 } } peers { permanent_uuid: "bb9261ea26ae46a580415794f87a5ee0" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 33863 } }
I20260620 20:19:35.554548 788 leader_election.cc:288] T adcc815954884850873c99f877af9be8 P bb9261ea26ae46a580415794f87a5ee0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 48ddf12e1ab742819abba58564f9d6b5 (127.16.217.1:38751), 24811d2b3eec4754a9194eabaa73b41f (127.16.217.3:46007)
I20260620 20:19:35.561743 776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:19:35.554978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.217.2:42189 (local address 127.16.217.1:38751)
0620 20:19:35.555188 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0620 20:19:35.555196 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 20:19:35.556112 (+ 916us) server_negotiation.cc:407] Connection header received
0620 20:19:35.556322 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:19:35.556327 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:19:35.556406 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:19:35.556521 (+ 115us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:19:35.557141 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:19:35.557874 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:19:35.559771 (+ 1897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:19:35.560746 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:19:35.560901 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:19:35.560994 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0620 20:19:35.561323 (+ 329us) server_negotiation.cc:299] Negotiation successful
0620 20:19:35.561464 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 20:19:35.561852 778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:19:35.555031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.217.2:58153 (local address 127.16.217.3:46007)
0620 20:19:35.555199 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0620 20:19:35.555215 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 20:19:35.555951 (+ 736us) server_negotiation.cc:407] Connection header received
0620 20:19:35.556111 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:19:35.556116 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:19:35.556216 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:19:35.556300 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:19:35.557018 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:19:35.557768 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:19:35.559762 (+ 1994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:19:35.560766 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:19:35.560948 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:19:35.561018 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 20:19:35.561468 (+ 450us) server_negotiation.cc:299] Negotiation successful
0620 20:19:35.561600 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 20:19:35.561977 791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:19:35.554820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.3:46007 (local address 127.16.217.2:58153)
0620 20:19:35.555833 (+ 1013us) negotiation.cc:107] Waiting for socket to connect
0620 20:19:35.555849 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 20:19:35.555974 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:19:35.556351 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:19:35.556356 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:19:35.556389 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:19:35.556849 (+ 460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:19:35.556862 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:19:35.557920 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:19:35.557926 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:19:35.559611 (+ 1685us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:19:35.559622 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:19:35.561003 (+ 1381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:19:35.561011 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:19:35.561205 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:19:35.561251 (+ 46us) client_negotiation.cc:769] Sending connection context
0620 20:19:35.561415 (+ 164us) client_negotiation.cc:240] Negotiation successful
0620 20:19:35.561526 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":911,"mutex_wait_us":5,"thread_start_us":688,"threads_started":1}
I20260620 20:19:35.562028 790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:19:35.554935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.1:38751 (local address 127.16.217.2:42189)
0620 20:19:35.555977 (+ 1042us) negotiation.cc:107] Waiting for socket to connect
0620 20:19:35.555995 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 20:19:35.556124 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:19:35.556557 (+ 433us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:19:35.556560 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:19:35.556579 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:19:35.556984 (+ 405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:19:35.556996 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:19:35.558030 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:19:35.558034 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:19:35.559603 (+ 1569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:19:35.559615 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:19:35.560964 (+ 1349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:19:35.560971 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:19:35.561155 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:19:35.561178 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 20:19:35.561258 (+ 80us) client_negotiation.cc:240] Negotiation successful
0620 20:19:35.561479 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":908,"mutex_wait_us":39,"thread_start_us":517,"threads_started":1}
I20260620 20:19:35.563007 718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "adcc815954884850873c99f877af9be8" candidate_uuid: "bb9261ea26ae46a580415794f87a5ee0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "24811d2b3eec4754a9194eabaa73b41f" is_pre_election: true
I20260620 20:19:35.563301 451 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "adcc815954884850873c99f877af9be8" candidate_uuid: "bb9261ea26ae46a580415794f87a5ee0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48ddf12e1ab742819abba58564f9d6b5" is_pre_election: true
I20260620 20:19:35.563460 718 raft_consensus.cc:2454] T adcc815954884850873c99f877af9be8 P 24811d2b3eec4754a9194eabaa73b41f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bb9261ea26ae46a580415794f87a5ee0 in term 0.
I20260620 20:19:35.563771 451 raft_consensus.cc:2454] T adcc815954884850873c99f877af9be8 P 48ddf12e1ab742819abba58564f9d6b5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bb9261ea26ae46a580415794f87a5ee0 in term 0.
I20260620 20:19:35.564076 519 leader_election.cc:302] T adcc815954884850873c99f877af9be8 P bb9261ea26ae46a580415794f87a5ee0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 24811d2b3eec4754a9194eabaa73b41f, bb9261ea26ae46a580415794f87a5ee0; no voters:
I20260620 20:19:35.564507 788 raft_consensus.cc:2781] T adcc815954884850873c99f877af9be8 P bb9261ea26ae46a580415794f87a5ee0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 20:19:35.564679 788 raft_consensus.cc:482] T adcc815954884850873c99f877af9be8 P bb9261ea26ae46a580415794f87a5ee0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 20:19:35.564795 788 raft_consensus.cc:3037] T adcc815954884850873c99f877af9be8 P bb9261ea26ae46a580415794f87a5ee0 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:19:35.566972 788 raft_consensus.cc:504] T adcc815954884850873c99f877af9be8 P bb9261ea26ae46a580415794f87a5ee0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48ddf12e1ab742819abba58564f9d6b5" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 38751 } } peers { permanent_uuid: "24811d2b3eec4754a9194eabaa73b41f" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 46007 } } peers { permanent_uuid: "bb9261ea26ae46a580415794f87a5ee0" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 33863 } }
I20260620 20:19:35.567570 788 leader_election.cc:288] T adcc815954884850873c99f877af9be8 P bb9261ea26ae46a580415794f87a5ee0 [CANDIDATE]: Term 1 election: Requested vote from peers 48ddf12e1ab742819abba58564f9d6b5 (127.16.217.1:38751), 24811d2b3eec4754a9194eabaa73b41f (127.16.217.3:46007)
I20260620 20:19:35.567955 451 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "adcc815954884850873c99f877af9be8" candidate_uuid: "bb9261ea26ae46a580415794f87a5ee0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48ddf12e1ab742819abba58564f9d6b5"
I20260620 20:19:35.568073 718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "adcc815954884850873c99f877af9be8" candidate_uuid: "bb9261ea26ae46a580415794f87a5ee0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "24811d2b3eec4754a9194eabaa73b41f"
I20260620 20:19:35.568147 451 raft_consensus.cc:3037] T adcc815954884850873c99f877af9be8 P 48ddf12e1ab742819abba58564f9d6b5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:19:35.568298 718 raft_consensus.cc:3037] T adcc815954884850873c99f877af9be8 P 24811d2b3eec4754a9194eabaa73b41f [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:19:35.570492 451 raft_consensus.cc:2454] T adcc815954884850873c99f877af9be8 P 48ddf12e1ab742819abba58564f9d6b5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb9261ea26ae46a580415794f87a5ee0 in term 1.
I20260620 20:19:35.570684 718 raft_consensus.cc:2454] T adcc815954884850873c99f877af9be8 P 24811d2b3eec4754a9194eabaa73b41f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb9261ea26ae46a580415794f87a5ee0 in term 1.
I20260620 20:19:35.570894 520 leader_election.cc:302] T adcc815954884850873c99f877af9be8 P bb9261ea26ae46a580415794f87a5ee0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48ddf12e1ab742819abba58564f9d6b5, bb9261ea26ae46a580415794f87a5ee0; no voters:
I20260620 20:19:35.571192 788 raft_consensus.cc:2781] T adcc815954884850873c99f877af9be8 P bb9261ea26ae46a580415794f87a5ee0 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:19:35.571676 788 raft_consensus.cc:686] T adcc815954884850873c99f877af9be8 P bb9261ea26ae46a580415794f87a5ee0 [term 1 LEADER]: Becoming Leader. State: Replica: bb9261ea26ae46a580415794f87a5ee0, State: Running, Role: LEADER
I20260620 20:19:35.572113 788 consensus_queue.cc:237] T adcc815954884850873c99f877af9be8 P bb9261ea26ae46a580415794f87a5ee0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48ddf12e1ab742819abba58564f9d6b5" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 38751 } } peers { permanent_uuid: "24811d2b3eec4754a9194eabaa73b41f" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 46007 } } peers { permanent_uuid: "bb9261ea26ae46a580415794f87a5ee0" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 33863 } }
I20260620 20:19:35.576932 307 catalog_manager.cc:5495] T adcc815954884850873c99f877af9be8 P bb9261ea26ae46a580415794f87a5ee0 reported cstate change: term changed from 0 to 1, leader changed from <none> to bb9261ea26ae46a580415794f87a5ee0 (127.16.217.2). New cstate: current_term: 1 leader_uuid: "bb9261ea26ae46a580415794f87a5ee0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48ddf12e1ab742819abba58564f9d6b5" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 38751 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "24811d2b3eec4754a9194eabaa73b41f" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 46007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bb9261ea26ae46a580415794f87a5ee0" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 33863 } health_report { overall_health: HEALTHY } } }
I20260620 20:19:35.686156 780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:19:35.676318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53340 (local address 127.16.217.2:33863)
0620 20:19:35.676668 (+ 350us) server_negotiation.cc:206] Beginning negotiation
0620 20:19:35.676676 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 20:19:35.677886 (+ 1210us) server_negotiation.cc:407] Connection header received
0620 20:19:35.678233 (+ 347us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:19:35.678237 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:19:35.678376 (+ 139us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:19:35.678511 (+ 135us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 20:19:35.679235 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:19:35.680398 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:19:35.682881 (+ 2483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:19:35.683717 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:19:35.683829 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:19:35.684695 (+ 866us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 20:19:35.685377 (+ 682us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 20:19:35.685548 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0620 20:19:35.685760 (+ 212us) server_negotiation.cc:299] Negotiation successful
0620 20:19:35.685843 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 20:19:35.697161 718 raft_consensus.cc:1261] T adcc815954884850873c99f877af9be8 P 24811d2b3eec4754a9194eabaa73b41f [term 1 FOLLOWER]: Refusing update from remote peer bb9261ea26ae46a580415794f87a5ee0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:19:35.697497 451 raft_consensus.cc:1261] T adcc815954884850873c99f877af9be8 P 48ddf12e1ab742819abba58564f9d6b5 [term 1 FOLLOWER]: Refusing update from remote peer bb9261ea26ae46a580415794f87a5ee0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:19:35.698208 788 consensus_queue.cc:1035] T adcc815954884850873c99f877af9be8 P bb9261ea26ae46a580415794f87a5ee0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "24811d2b3eec4754a9194eabaa73b41f" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 46007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:19:35.698522 792 consensus_queue.cc:1035] T adcc815954884850873c99f877af9be8 P bb9261ea26ae46a580415794f87a5ee0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48ddf12e1ab742819abba58564f9d6b5" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 38751 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:19:35.714633 796 mvcc.cc:205] Tried to move back new op lower bound from 7299017833242898432 to 7299017832747175936. Current Snapshot: MvccSnapshot[applied={T|T < 7299017833242898432 or (T in {7299017833242898432})}]
I20260620 20:19:35.714993 798 mvcc.cc:205] Tried to move back new op lower bound from 7299017833242898432 to 7299017832747175936. Current Snapshot: MvccSnapshot[applied={T|T < 7299017833242898432 or (T in {7299017833242898432})}]
I20260620 20:19:35.728137 307 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43424:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 20:19:35.728396 307 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43424:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 20:19:35.731986 307 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8e39d1abbd4e46a7bc73a63bfb83dfd0: Sending DeleteTablet for 3 replicas of tablet adcc815954884850873c99f877af9be8
I20260620 20:19:35.733207 431 tablet_service.cc:1514] Processing DeleteTablet for tablet adcc815954884850873c99f877af9be8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:19:35 UTC) from {username='kudu', principal='kudu/127.16.217.62@KRBTEST.COM'} at 127.0.0.1:32876
I20260620 20:19:35.733397 17252 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu pid 368
I20260620 20:19:35.733419 698 tablet_service.cc:1514] Processing DeleteTablet for tablet adcc815954884850873c99f877af9be8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:19:35 UTC) from {username='kudu', principal='kudu/127.16.217.62@KRBTEST.COM'} at 127.0.0.1:47892
I20260620 20:19:35.734108 565 tablet_service.cc:1514] Processing DeleteTablet for tablet adcc815954884850873c99f877af9be8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:19:35 UTC) from {username='kudu', principal='kudu/127.16.217.62@KRBTEST.COM'} at 127.0.0.1:53338
I20260620 20:19:35.742393 808 tablet_replica.cc:330] T adcc815954884850873c99f877af9be8 P bb9261ea26ae46a580415794f87a5ee0: stopping tablet replica
I20260620 20:19:35.742938 808 raft_consensus.cc:2229] T adcc815954884850873c99f877af9be8 P bb9261ea26ae46a580415794f87a5ee0 [term 1 LEADER]: Raft consensus shutting down.
I20260620 20:19:35.743706 808 raft_consensus.cc:2258] T adcc815954884850873c99f877af9be8 P bb9261ea26ae46a580415794f87a5ee0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:19:35.745380 805 tablet_replica.cc:330] T adcc815954884850873c99f877af9be8 P 48ddf12e1ab742819abba58564f9d6b5: stopping tablet replica
I20260620 20:19:35.745832 805 raft_consensus.cc:2229] T adcc815954884850873c99f877af9be8 P 48ddf12e1ab742819abba58564f9d6b5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 20:19:35.746376 805 raft_consensus.cc:2258] T adcc815954884850873c99f877af9be8 P 48ddf12e1ab742819abba58564f9d6b5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:19:35.746642 808 ts_tablet_manager.cc:1905] T adcc815954884850873c99f877af9be8 P bb9261ea26ae46a580415794f87a5ee0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:19:35.748643 805 ts_tablet_manager.cc:1905] T adcc815954884850873c99f877af9be8 P 48ddf12e1ab742819abba58564f9d6b5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:19:35.749161 806 tablet_replica.cc:330] T adcc815954884850873c99f877af9be8 P 24811d2b3eec4754a9194eabaa73b41f: stopping tablet replica
I20260620 20:19:35.749611 806 raft_consensus.cc:2229] T adcc815954884850873c99f877af9be8 P 24811d2b3eec4754a9194eabaa73b41f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 20:19:35.750133 806 raft_consensus.cc:2258] T adcc815954884850873c99f877af9be8 P 24811d2b3eec4754a9194eabaa73b41f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:19:35.752045 806 ts_tablet_manager.cc:1905] T adcc815954884850873c99f877af9be8 P 24811d2b3eec4754a9194eabaa73b41f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:19:35.754771 808 ts_tablet_manager.cc:1918] T adcc815954884850873c99f877af9be8 P bb9261ea26ae46a580415794f87a5ee0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 20:19:35.754944 808 log.cc:1196] T adcc815954884850873c99f877af9be8 P bb9261ea26ae46a580415794f87a5ee0: Deleting WAL directory at /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986536687152-17252-0/minicluster-data/ts-1/wal/wals/adcc815954884850873c99f877af9be8
I20260620 20:19:35.755465 808 ts_tablet_manager.cc:1939] T adcc815954884850873c99f877af9be8 P bb9261ea26ae46a580415794f87a5ee0: Deleting consensus metadata
I20260620 20:19:35.756204 805 ts_tablet_manager.cc:1918] T adcc815954884850873c99f877af9be8 P 48ddf12e1ab742819abba58564f9d6b5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 20:19:35.756367 805 log.cc:1196] T adcc815954884850873c99f877af9be8 P 48ddf12e1ab742819abba58564f9d6b5: Deleting WAL directory at /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986536687152-17252-0/minicluster-data/ts-0/wal/wals/adcc815954884850873c99f877af9be8
Jun 20 20:19:35 dist-test-slave-2hpg krb5kdc[32730](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986772, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.217.1@KRBTEST.COM
I20260620 20:19:35.756875 805 ts_tablet_manager.cc:1939] T adcc815954884850873c99f877af9be8 P 48ddf12e1ab742819abba58564f9d6b5: Deleting consensus metadata
I20260620 20:19:35.757215 32761 catalog_manager.cc:4841] TS bb9261ea26ae46a580415794f87a5ee0 (127.16.217.2:33863): tablet adcc815954884850873c99f877af9be8 (table test-table [id=a5b5c8c4f0d646888a59c37ea3a0a6dc]) successfully deleted
I20260620 20:19:35.759377 806 ts_tablet_manager.cc:1918] T adcc815954884850873c99f877af9be8 P 24811d2b3eec4754a9194eabaa73b41f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 20:19:35.759531 806 log.cc:1196] T adcc815954884850873c99f877af9be8 P 24811d2b3eec4754a9194eabaa73b41f: Deleting WAL directory at /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986536687152-17252-0/minicluster-data/ts-2/wal/wals/adcc815954884850873c99f877af9be8
I20260620 20:19:35.759498 32761 catalog_manager.cc:4841] TS 48ddf12e1ab742819abba58564f9d6b5 (127.16.217.1:38751): tablet adcc815954884850873c99f877af9be8 (table test-table [id=a5b5c8c4f0d646888a59c37ea3a0a6dc]) successfully deleted
I20260620 20:19:35.760028 806 ts_tablet_manager.cc:1939] T adcc815954884850873c99f877af9be8 P 24811d2b3eec4754a9194eabaa73b41f: Deleting consensus metadata
I20260620 20:19:35.761914 32760 catalog_manager.cc:4841] TS 24811d2b3eec4754a9194eabaa73b41f (127.16.217.3:46007): tablet adcc815954884850873c99f877af9be8 (table test-table [id=a5b5c8c4f0d646888a59c37ea3a0a6dc]) successfully deleted
I20260620 20:19:35.765067 776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:19:35.734683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32886 (local address 127.16.217.1:38751)
0620 20:19:35.734858 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0620 20:19:35.734864 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:19:35.744612 (+ 9748us) server_negotiation.cc:407] Connection header received
0620 20:19:35.744826 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:19:35.744831 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:19:35.744921 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:19:35.745026 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:19:35.747775 (+ 2749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:19:35.749151 (+ 1376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:19:35.751421 (+ 2270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:19:35.752086 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:19:35.752189 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:19:35.758361 (+ 6172us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:19:35.758376 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:19:35.758394 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:19:35.758439 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:19:35.762309 (+ 3870us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:19:35.763021 (+ 712us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:19:35.763028 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:19:35.763031 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:19:35.763153 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:19:35.763534 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:19:35.763543 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:19:35.763545 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:19:35.763892 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:19:35.764193 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0620 20:19:35.764510 (+ 317us) server_negotiation.cc:299] Negotiation successful
0620 20:19:35.764727 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 20:19:35.765756 491 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:19:35.875254 17252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu with pid 368
I20260620 20:19:35.892900 17252 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu pid 502
Jun 20 20:19:35 dist-test-slave-2hpg krb5kdc[32730](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986772, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.217.2@KRBTEST.COM
I20260620 20:19:35.910728 780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:19:35.893682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53348 (local address 127.16.217.2:33863)
0620 20:19:35.893902 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0620 20:19:35.893908 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:19:35.894038 (+ 130us) server_negotiation.cc:407] Connection header received
0620 20:19:35.894199 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:19:35.894205 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:19:35.894315 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:19:35.894427 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:19:35.895707 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:19:35.896953 (+ 1246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:19:35.898401 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:19:35.899066 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:19:35.899201 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:19:35.903599 (+ 4398us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:19:35.903618 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:19:35.903644 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:19:35.903698 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:19:35.907668 (+ 3970us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:19:35.908379 (+ 711us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:19:35.908388 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:19:35.908392 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:19:35.908570 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:19:35.908980 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:19:35.908984 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:19:35.908987 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:19:35.909416 (+ 429us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:19:35.909756 (+ 340us) server_negotiation.cc:1035] Waiting for connection context
0620 20:19:35.910087 (+ 331us) server_negotiation.cc:299] Negotiation successful
0620 20:19:35.910320 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 20:19:35.911535 625 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:19:36.023326 17252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu with pid 502
I20260620 20:19:36.041838 17252 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu pid 635
Jun 20 20:19:36 dist-test-slave-2hpg krb5kdc[32730](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986772, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.217.3@KRBTEST.COM
I20260620 20:19:36.058090 778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:19:36.042674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47900 (local address 127.16.217.3:46007)
0620 20:19:36.042846 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0620 20:19:36.042850 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:19:36.042914 (+ 64us) server_negotiation.cc:407] Connection header received
0620 20:19:36.043065 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:19:36.043069 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:19:36.043142 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:19:36.043229 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:19:36.044784 (+ 1555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:19:36.045719 (+ 935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:19:36.047066 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:19:36.047446 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:19:36.047544 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:19:36.052290 (+ 4746us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:19:36.052307 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:19:36.052323 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:19:36.052362 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:19:36.055405 (+ 3043us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:19:36.056104 (+ 699us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:19:36.056113 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:19:36.056115 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:19:36.056295 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:19:36.056657 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:19:36.056661 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:19:36.056664 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:19:36.057032 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:19:36.057264 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0620 20:19:36.057531 (+ 267us) server_negotiation.cc:299] Negotiation successful
0620 20:19:36.057693 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 20:19:36.058748 758 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:19:36.146579 17252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu with pid 635
I20260620 20:19:36.163419 17252 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu pid 32743
I20260620 20:19:36.164575 337 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:19:36.275099 17252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu with pid 32743
I20260620 20:19:56.296639 17252 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
/tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986536687152-17252-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986536687152-17252-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986536687152-17252-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986536687152-17252-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.217.62:34019
--webserver_interface=127.16.217.62
--webserver_port=35859
--keytab_file=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.62.keytab
--principal=kudu/127.16.217.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.217.20:35087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.217.62:34019
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986536687152-17252-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986536687152-17252-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986536687152-17252-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:19:56.640506 813 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:19:56.640856 813 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:19:56.641017 813 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:19:56.648463 813 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260620 20:19:56.648600 813 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260620 20:19:56.648691 813 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260620 20:19:56.648773 813 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:19:56.648871 813 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:19:56.648986 813 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:19:56.649070 813 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:19:56.649160 813 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:19:56.659477 813 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.217.20:35087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986536687152-17252-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986536687152-17252-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.16.217.62:34019
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.62.keytab
--principal=kudu/127.16.217.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.217.62:34019
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986536687152-17252-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.217.62
--webserver_port=35859
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986536687152-17252-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:19:56.660717 813 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:19:56.662021 813 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:19:56.686785 820 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:19:56.686769 822 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:19:56.686769 819 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:19:56.687614 813 server_base.cc:1034] running on GCE node
I20260620 20:19:56.688462 813 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:19:56.690078 813 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:19:56.691341 813 hybrid_clock.cc:648] HybridClock initialized: now 1781986796691257 us; error 43 us; skew 500 ppm
I20260620 20:19:56.691834 813 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:19:56 dist-test-slave-2hpg krb5kdc[32730](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986796, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.217.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:19:56.695926 813 init.cc:377] Logged in from keytab as kudu/127.16.217.62@KRBTEST.COM (short username kudu)
I20260620 20:19:56.697674 813 webserver.cc:460] Webserver started at http://127.16.217.62:35859/ using document root <none> and password file <none>
I20260620 20:19:56.698536 813 fs_manager.cc:359] Metadata directory not provided
I20260620 20:19:56.698691 813 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:19:56.705360 813 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260620 20:19:56.708734 829 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:19:56.710618 813 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260620 20:19:56.710837 813 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986536687152-17252-0/minicluster-data/master-0/data,/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986536687152-17252-0/minicluster-data/master-0/wal
uuid: "8e39d1abbd4e46a7bc73a63bfb83dfd0"
format_stamp: "Formatted at 2026-06-20 20:19:33 on dist-test-slave-2hpg"
server_key: "e31576e9a31de7cbfbaa3f0936591892"
server_key_iv: "1cea50ed55f4321d6496cb1ca04baf58"
server_key_version: "encryptionkey@0"
I20260620 20:19:56.712390 813 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986536687152-17252-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986536687152-17252-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986536687152-17252-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:19:56.736876 813 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:19:56.741737 813 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:19:56.742097 813 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:19:56.762727 813 rpc_server.cc:307] RPC server started. Bound to: 127.16.217.62:34019
I20260620 20:19:56.762928 880 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.217.62:34019 every 8 connection(s)
I20260620 20:19:56.764312 813 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986536687152-17252-0/minicluster-data/master-0/data/info.pb
I20260620 20:19:56.771718 881 sys_catalog.cc:263] Verifying existing consensus state
I20260620 20:19:56.771899 17252 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu as pid 813
I20260620 20:19:56.772871 17252 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
/tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986536687152-17252-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986536687152-17252-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986536687152-17252-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986536687152-17252-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.217.1:38751
--local_ip_for_outbound_sockets=127.16.217.1
--tserver_master_addrs=127.16.217.62:34019
--webserver_port=46685
--webserver_interface=127.16.217.1
--keytab_file=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.1.keytab
--principal=kudu/127.16.217.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.217.20:35087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986536687152-17252-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986536687152-17252-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986536687152-17252-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 20:19:56.777034 881 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8e39d1abbd4e46a7bc73a63bfb83dfd0: Bootstrap starting.
I20260620 20:19:56.798950 881 log.cc:824] T 00000000000000000000000000000000 P 8e39d1abbd4e46a7bc73a63bfb83dfd0: Log is configured to *not* fsync() on all Append() calls
I20260620 20:19:56.811641 881 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8e39d1abbd4e46a7bc73a63bfb83dfd0: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 20:19:56.812297 881 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8e39d1abbd4e46a7bc73a63bfb83dfd0: Bootstrap complete.
I20260620 20:19:56.822077 881 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8e39d1abbd4e46a7bc73a63bfb83dfd0 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e39d1abbd4e46a7bc73a63bfb83dfd0" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 34019 } }
I20260620 20:19:56.822932 881 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8e39d1abbd4e46a7bc73a63bfb83dfd0 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e39d1abbd4e46a7bc73a63bfb83dfd0, State: Initialized, Role: FOLLOWER
I20260620 20:19:56.823635 881 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8e39d1abbd4e46a7bc73a63bfb83dfd0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e39d1abbd4e46a7bc73a63bfb83dfd0" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 34019 } }
I20260620 20:19:56.823889 881 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8e39d1abbd4e46a7bc73a63bfb83dfd0 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 20:19:56.824008 881 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8e39d1abbd4e46a7bc73a63bfb83dfd0 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 20:19:56.824231 881 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8e39d1abbd4e46a7bc73a63bfb83dfd0 [term 1 FOLLOWER]: Advancing to term 2
I20260620 20:19:56.828473 881 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8e39d1abbd4e46a7bc73a63bfb83dfd0 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e39d1abbd4e46a7bc73a63bfb83dfd0" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 34019 } }
I20260620 20:19:56.829063 881 leader_election.cc:302] T 00000000000000000000000000000000 P 8e39d1abbd4e46a7bc73a63bfb83dfd0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8e39d1abbd4e46a7bc73a63bfb83dfd0; no voters:
I20260620 20:19:56.829761 881 leader_election.cc:288] T 00000000000000000000000000000000 P 8e39d1abbd4e46a7bc73a63bfb83dfd0 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 20:19:56.829916 885 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8e39d1abbd4e46a7bc73a63bfb83dfd0 [term 2 FOLLOWER]: Leader election won for term 2
I20260620 20:19:56.831074 885 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8e39d1abbd4e46a7bc73a63bfb83dfd0 [term 2 LEADER]: Becoming Leader. State: Replica: 8e39d1abbd4e46a7bc73a63bfb83dfd0, State: Running, Role: LEADER
I20260620 20:19:56.831743 885 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8e39d1abbd4e46a7bc73a63bfb83dfd0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e39d1abbd4e46a7bc73a63bfb83dfd0" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 34019 } }
I20260620 20:19:56.832190 881 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8e39d1abbd4e46a7bc73a63bfb83dfd0 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:19:56.834534 887 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8e39d1abbd4e46a7bc73a63bfb83dfd0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8e39d1abbd4e46a7bc73a63bfb83dfd0. Latest consensus state: current_term: 2 leader_uuid: "8e39d1abbd4e46a7bc73a63bfb83dfd0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e39d1abbd4e46a7bc73a63bfb83dfd0" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 34019 } } }
I20260620 20:19:56.835005 887 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8e39d1abbd4e46a7bc73a63bfb83dfd0 [sys.catalog]: This master's current role is: LEADER
I20260620 20:19:56.835543 886 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8e39d1abbd4e46a7bc73a63bfb83dfd0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "8e39d1abbd4e46a7bc73a63bfb83dfd0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e39d1abbd4e46a7bc73a63bfb83dfd0" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 34019 } } }
I20260620 20:19:56.835913 886 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8e39d1abbd4e46a7bc73a63bfb83dfd0 [sys.catalog]: This master's current role is: LEADER
I20260620 20:19:56.838042 891 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 20:19:56.847558 891 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 20:19:56.849431 891 catalog_manager.cc:1259] Loaded cluster ID: e0dabc224e8647618b3b459dc9e53e35
I20260620 20:19:56.849611 891 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 20:19:56.854859 891 catalog_manager.cc:1504] Loading token signing keys...
I20260620 20:19:56.856894 891 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 8e39d1abbd4e46a7bc73a63bfb83dfd0: Loaded TSK: 0
I20260620 20:19:56.913424 891 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8e39d1abbd4e46a7bc73a63bfb83dfd0: Generated new TSK 1
I20260620 20:19:56.913990 891 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 20:19:57.156028 883 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:19:57.156384 883 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:19:57.156549 883 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:19:57.164055 883 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:19:57.164222 883 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:19:57.164400 883 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.217.1
I20260620 20:19:57.174660 883 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.217.20:35087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986536687152-17252-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986536687152-17252-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.1.keytab
--principal=kudu/127.16.217.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.217.1:38751
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986536687152-17252-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.217.1
--webserver_port=46685
--tserver_master_addrs=127.16.217.62:34019
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.217.1
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986536687152-17252-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:19:57.176000 883 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:19:57.177399 883 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:19:57.193857 909 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:19:57.194150 912 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:19:57.193970 910 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:19:57.196477 883 server_base.cc:1034] running on GCE node
I20260620 20:19:57.197384 883 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:19:57.198881 883 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:19:57.202320 883 hybrid_clock.cc:648] HybridClock initialized: now 1781986797202255 us; error 42 us; skew 500 ppm
I20260620 20:19:57.202692 883 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:19:57 dist-test-slave-2hpg krb5kdc[32730](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986797, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.217.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:19:57.206105 883 init.cc:377] Logged in from keytab as kudu/127.16.217.1@KRBTEST.COM (short username kudu)
I20260620 20:19:57.207811 883 webserver.cc:460] Webserver started at http://127.16.217.1:46685/ using document root <none> and password file <none>
I20260620 20:19:57.208636 883 fs_manager.cc:359] Metadata directory not provided
I20260620 20:19:57.208752 883 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:19:57.214393 883 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260620 20:19:57.217317 919 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:19:57.218911 883 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 20:19:57.219130 883 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986536687152-17252-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986536687152-17252-0/minicluster-data/ts-0/wal
uuid: "48ddf12e1ab742819abba58564f9d6b5"
format_stamp: "Formatted at 2026-06-20 20:19:34 on dist-test-slave-2hpg"
server_key: "a59ce49777766b69fa009b135592f350"
server_key_iv: "f28fb0e2561f6d706f9cf30682c90572"
server_key_version: "encryptionkey@0"
I20260620 20:19:57.220155 883 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986536687152-17252-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986536687152-17252-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986536687152-17252-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:19:57.245227 883 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:19:57.249852 883 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:19:57.250197 883 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:19:57.251194 883 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:19:57.252656 883 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:19:57.252784 883 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:19:57.252905 883 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:19:57.252988 883 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:19:57.288271 883 rpc_server.cc:307] RPC server started. Bound to: 127.16.217.1:38751
I20260620 20:19:57.288365 1031 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.217.1:38751 every 8 connection(s)
I20260620 20:19:57.289749 883 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986536687152-17252-0/minicluster-data/ts-0/data/info.pb
I20260620 20:19:57.290941 17252 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu as pid 883
I20260620 20:19:57.291697 17252 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
/tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986536687152-17252-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986536687152-17252-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986536687152-17252-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986536687152-17252-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.217.2:33863
--local_ip_for_outbound_sockets=127.16.217.2
--tserver_master_addrs=127.16.217.62:34019
--webserver_port=44433
--webserver_interface=127.16.217.2
--keytab_file=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.2.keytab
--principal=kudu/127.16.217.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.217.20:35087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986536687152-17252-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986536687152-17252-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986536687152-17252-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 20:19:57 dist-test-slave-2hpg krb5kdc[32730](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986797, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.217.1@KRBTEST.COM for kudu/127.16.217.62@KRBTEST.COM
I20260620 20:19:57.317662 1035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:19:57.293477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:34019 (local address 127.16.217.1:33933)
0620 20:19:57.294806 (+ 1329us) negotiation.cc:107] Waiting for socket to connect
0620 20:19:57.294855 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0620 20:19:57.297696 (+ 2841us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:19:57.299385 (+ 1689us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:19:57.299398 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:19:57.300085 (+ 687us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:19:57.300661 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:19:57.300680 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:19:57.302422 (+ 1742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:19:57.302428 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:19:57.303725 (+ 1297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:19:57.303737 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:19:57.304864 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:19:57.304872 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:19:57.305173 (+ 301us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:19:57.306078 (+ 905us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:19:57.306104 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:19:57.308687 (+ 2583us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:19:57.312657 (+ 3970us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:19:57.312670 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:19:57.312693 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:19:57.313209 (+ 516us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:19:57.313715 (+ 506us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:19:57.313721 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:19:57.313725 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:19:57.314017 (+ 292us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:19:57.314857 (+ 840us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:19:57.314866 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:19:57.315255 (+ 389us) client_negotiation.cc:769] Sending connection context
0620 20:19:57.315554 (+ 299us) client_negotiation.cc:240] Negotiation successful
0620 20:19:57.316176 (+ 622us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1019,"thread_start_us":806,"threads_started":1}
I20260620 20:19:57.317679 1034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:19:57.293345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.217.1:33933 (local address 127.16.217.62:34019)
0620 20:19:57.294166 (+ 821us) server_negotiation.cc:206] Beginning negotiation
0620 20:19:57.294185 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0620 20:19:57.295026 (+ 841us) server_negotiation.cc:407] Connection header received
0620 20:19:57.298428 (+ 3402us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:19:57.298443 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:19:57.298829 (+ 386us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:19:57.299166 (+ 337us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:19:57.300910 (+ 1744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:19:57.302258 (+ 1348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:19:57.303888 (+ 1630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:19:57.304590 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:19:57.304828 (+ 238us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:19:57.309016 (+ 4188us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:19:57.309026 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:19:57.309043 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:19:57.309084 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:19:57.312464 (+ 3380us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:19:57.313386 (+ 922us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:19:57.313394 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:19:57.313400 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:19:57.313575 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:19:57.314139 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:19:57.314146 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:19:57.314152 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:19:57.314698 (+ 546us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:19:57.314923 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0620 20:19:57.315832 (+ 909us) server_negotiation.cc:299] Negotiation successful
0620 20:19:57.316261 (+ 429us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":508,"thread_start_us":315,"threads_started":1}
I20260620 20:19:57.320741 1032 heartbeater.cc:347] Connected to a master server at 127.16.217.62:34019
I20260620 20:19:57.321082 1032 heartbeater.cc:464] Registering TS with master...
I20260620 20:19:57.321923 1032 heartbeater.cc:511] Master 127.16.217.62:34019 requested a full tablet report, sending...
I20260620 20:19:57.324359 846 ts_manager.cc:194] Registered new tserver with Master: 48ddf12e1ab742819abba58564f9d6b5 (127.16.217.1:38751)
I20260620 20:19:57.326103 846 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.217.1@KRBTEST.COM'} at 127.16.217.1:33933
W20260620 20:19:57.631969 1036 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:19:57.632310 1036 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:19:57.632477 1036 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:19:57.640048 1036 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:19:57.640202 1036 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:19:57.640399 1036 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.217.2
I20260620 20:19:57.650149 1036 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.217.20:35087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986536687152-17252-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986536687152-17252-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.2.keytab
--principal=kudu/127.16.217.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.217.2:33863
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986536687152-17252-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.217.2
--webserver_port=44433
--tserver_master_addrs=127.16.217.62:34019
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.217.2
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986536687152-17252-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:19:57.651448 1036 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:19:57.652841 1036 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:19:57.666620 1041 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:19:57.666647 1042 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:19:57.666880 1044 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:19:57.667511 1036 server_base.cc:1034] running on GCE node
I20260620 20:19:57.668172 1036 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:19:57.669250 1036 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:19:57.670537 1036 hybrid_clock.cc:648] HybridClock initialized: now 1781986797670499 us; error 62 us; skew 500 ppm
I20260620 20:19:57.670962 1036 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:19:57 dist-test-slave-2hpg krb5kdc[32730](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986797, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.217.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:19:57.674290 1036 init.cc:377] Logged in from keytab as kudu/127.16.217.2@KRBTEST.COM (short username kudu)
I20260620 20:19:57.675800 1036 webserver.cc:460] Webserver started at http://127.16.217.2:44433/ using document root <none> and password file <none>
I20260620 20:19:57.676584 1036 fs_manager.cc:359] Metadata directory not provided
I20260620 20:19:57.676746 1036 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:19:57.682430 1036 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260620 20:19:57.685253 1051 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:19:57.686748 1036 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 20:19:57.686910 1036 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986536687152-17252-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986536687152-17252-0/minicluster-data/ts-1/wal
uuid: "bb9261ea26ae46a580415794f87a5ee0"
format_stamp: "Formatted at 2026-06-20 20:19:34 on dist-test-slave-2hpg"
server_key: "7b9d4c4e8c73a23e07f972a525e02b22"
server_key_iv: "caa256f0c4f6f0763ef305f7f306aecd"
server_key_version: "encryptionkey@0"
I20260620 20:19:57.687916 1036 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986536687152-17252-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986536687152-17252-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986536687152-17252-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:19:57.706840 1036 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:19:57.711402 1036 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:19:57.711721 1036 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:19:57.712786 1036 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:19:57.714537 1036 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:19:57.714664 1036 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:19:57.714779 1036 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:19:57.714866 1036 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:19:57.749965 1036 rpc_server.cc:307] RPC server started. Bound to: 127.16.217.2:33863
I20260620 20:19:57.750155 1163 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.217.2:33863 every 8 connection(s)
I20260620 20:19:57.751400 1036 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986536687152-17252-0/minicluster-data/ts-1/data/info.pb
I20260620 20:19:57.758997 17252 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu as pid 1036
I20260620 20:19:57.759614 17252 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
/tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986536687152-17252-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986536687152-17252-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986536687152-17252-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986536687152-17252-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.217.3:46007
--local_ip_for_outbound_sockets=127.16.217.3
--tserver_master_addrs=127.16.217.62:34019
--webserver_port=38299
--webserver_interface=127.16.217.3
--keytab_file=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.3.keytab
--principal=kudu/127.16.217.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.217.20:35087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986536687152-17252-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986536687152-17252-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986536687152-17252-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 20:19:57 dist-test-slave-2hpg krb5kdc[32730](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986797, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.217.2@KRBTEST.COM for kudu/127.16.217.62@KRBTEST.COM
I20260620 20:19:57.773262 1034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:19:57.754127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.217.2:56085 (local address 127.16.217.62:34019)
0620 20:19:57.754340 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0620 20:19:57.754346 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:19:57.755582 (+ 1236us) server_negotiation.cc:407] Connection header received
0620 20:19:57.757388 (+ 1806us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:19:57.757393 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:19:57.757474 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:19:57.757562 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:19:57.759065 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:19:57.759927 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:19:57.761476 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:19:57.762013 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:19:57.762161 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:19:57.767069 (+ 4908us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:19:57.767079 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:19:57.767086 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:19:57.767142 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:19:57.770240 (+ 3098us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:19:57.770987 (+ 747us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:19:57.770992 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:19:57.770995 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:19:57.771126 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:19:57.771543 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:19:57.771548 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:19:57.771551 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:19:57.771849 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:19:57.771998 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0620 20:19:57.772691 (+ 693us) server_negotiation.cc:299] Negotiation successful
0620 20:19:57.772912 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260620 20:19:57.774353 1166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:19:57.754691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:34019 (local address 127.16.217.2:56085)
0620 20:19:57.755412 (+ 721us) negotiation.cc:107] Waiting for socket to connect
0620 20:19:57.755461 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0620 20:19:57.757133 (+ 1672us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:19:57.757801 (+ 668us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:19:57.757809 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:19:57.758327 (+ 518us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:19:57.758881 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:19:57.758897 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:19:57.760092 (+ 1195us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:19:57.760098 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:19:57.761326 (+ 1228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:19:57.761337 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:19:57.762261 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:19:57.762271 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:19:57.762610 (+ 339us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:19:57.763430 (+ 820us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:19:57.763471 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:19:57.766770 (+ 3299us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:19:57.770381 (+ 3611us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:19:57.770390 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:19:57.770406 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:19:57.770855 (+ 449us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:19:57.771233 (+ 378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:19:57.771238 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:19:57.771242 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:19:57.771427 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:19:57.771981 (+ 554us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:19:57.771989 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:19:57.772305 (+ 316us) client_negotiation.cc:769] Sending connection context
0620 20:19:57.772611 (+ 306us) client_negotiation.cc:240] Negotiation successful
0620 20:19:57.773111 (+ 500us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":472,"thread_start_us":285,"threads_started":1}
I20260620 20:19:57.775813 1164 heartbeater.cc:347] Connected to a master server at 127.16.217.62:34019
I20260620 20:19:57.776175 1164 heartbeater.cc:464] Registering TS with master...
I20260620 20:19:57.777145 1164 heartbeater.cc:511] Master 127.16.217.62:34019 requested a full tablet report, sending...
I20260620 20:19:57.778800 846 ts_manager.cc:194] Registered new tserver with Master: bb9261ea26ae46a580415794f87a5ee0 (127.16.217.2:33863)
I20260620 20:19:57.779933 846 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.217.2@KRBTEST.COM'} at 127.16.217.2:56085
W20260620 20:19:58.098158 1167 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:19:58.098438 1167 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:19:58.098645 1167 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:19:58.106163 1167 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:19:58.106305 1167 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:19:58.106585 1167 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.217.3
I20260620 20:19:58.116901 1167 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.217.20:35087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986536687152-17252-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986536687152-17252-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.3.keytab
--principal=kudu/127.16.217.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.217.3:46007
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986536687152-17252-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.217.3
--webserver_port=38299
--tserver_master_addrs=127.16.217.62:34019
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.217.3
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986536687152-17252-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:19:58.118314 1167 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:19:58.119800 1167 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:19:58.133567 1172 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:19:58.133633 1173 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:19:58.133612 1175 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:19:58.138527 1167 server_base.cc:1034] running on GCE node
I20260620 20:19:58.139210 1167 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:19:58.140228 1167 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:19:58.141477 1167 hybrid_clock.cc:648] HybridClock initialized: now 1781986798141425 us; error 49 us; skew 500 ppm
I20260620 20:19:58.141841 1167 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:19:58 dist-test-slave-2hpg krb5kdc[32730](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986798, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.217.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:19:58.145272 1167 init.cc:377] Logged in from keytab as kudu/127.16.217.3@KRBTEST.COM (short username kudu)
I20260620 20:19:58.146978 1167 webserver.cc:460] Webserver started at http://127.16.217.3:38299/ using document root <none> and password file <none>
I20260620 20:19:58.147786 1167 fs_manager.cc:359] Metadata directory not provided
I20260620 20:19:58.147946 1167 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:19:58.153733 1167 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260620 20:19:58.156617 1182 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:19:58.158181 1167 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 20:19:58.158349 1167 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986536687152-17252-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986536687152-17252-0/minicluster-data/ts-2/wal
uuid: "24811d2b3eec4754a9194eabaa73b41f"
format_stamp: "Formatted at 2026-06-20 20:19:35 on dist-test-slave-2hpg"
server_key: "4d2eb5a7b0d6dc44451589d8f4e00b44"
server_key_iv: "cf7eb6dbad246eff8cf648f969b186d1"
server_key_version: "encryptionkey@0"
I20260620 20:19:58.159319 1167 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986536687152-17252-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986536687152-17252-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986536687152-17252-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:19:58.176622 1167 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:19:58.180899 1167 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:19:58.181206 1167 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:19:58.182319 1167 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:19:58.183765 1167 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:19:58.183900 1167 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:19:58.184051 1167 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:19:58.184161 1167 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:19:58.219246 1167 rpc_server.cc:307] RPC server started. Bound to: 127.16.217.3:46007
I20260620 20:19:58.219332 1294 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.217.3:46007 every 8 connection(s)
I20260620 20:19:58.220715 1167 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986536687152-17252-0/minicluster-data/ts-2/data/info.pb
I20260620 20:19:58.228947 17252 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu as pid 1167
Jun 20 20:19:58 dist-test-slave-2hpg krb5kdc[32730](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986798, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.217.3@KRBTEST.COM for kudu/127.16.217.62@KRBTEST.COM
I20260620 20:19:58.243944 1034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:19:58.223424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.217.3:46563 (local address 127.16.217.62:34019)
0620 20:19:58.223701 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0620 20:19:58.223708 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:19:58.225083 (+ 1375us) server_negotiation.cc:407] Connection header received
0620 20:19:58.227241 (+ 2158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:19:58.227246 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:19:58.227316 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:19:58.227402 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:19:58.229175 (+ 1773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:19:58.230300 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:19:58.232241 (+ 1941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:19:58.233067 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:19:58.233198 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:19:58.237110 (+ 3912us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:19:58.237115 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:19:58.237118 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:19:58.237148 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:19:58.240726 (+ 3578us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:19:58.241432 (+ 706us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:19:58.241439 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:19:58.241443 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:19:58.241594 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:19:58.242066 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:19:58.242073 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:19:58.242078 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:19:58.242447 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:19:58.242640 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0620 20:19:58.243363 (+ 723us) server_negotiation.cc:299] Negotiation successful
0620 20:19:58.243512 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":27,"server-negotiator.queue_time_us":72}
I20260620 20:19:58.245272 1297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:19:58.224054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:34019 (local address 127.16.217.3:46563)
0620 20:19:58.224910 (+ 856us) negotiation.cc:107] Waiting for socket to connect
0620 20:19:58.224964 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0620 20:19:58.226942 (+ 1978us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:19:58.227626 (+ 684us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:19:58.227634 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:19:58.228178 (+ 544us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:19:58.228966 (+ 788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:19:58.228994 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:19:58.230539 (+ 1545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:19:58.230546 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:19:58.232079 (+ 1533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:19:58.232091 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:19:58.233272 (+ 1181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:19:58.233277 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:19:58.233481 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:19:58.234286 (+ 805us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:19:58.234309 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:19:58.236875 (+ 2566us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:19:58.240864 (+ 3989us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:19:58.240871 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:19:58.240884 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:19:58.241300 (+ 416us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:19:58.241713 (+ 413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:19:58.241716 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:19:58.241718 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:19:58.241961 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:19:58.242581 (+ 620us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:19:58.242587 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:19:58.242961 (+ 374us) client_negotiation.cc:769] Sending connection context
0620 20:19:58.243279 (+ 318us) client_negotiation.cc:240] Negotiation successful
0620 20:19:58.243776 (+ 497us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":466,"thread_start_us":274,"threads_started":1}
I20260620 20:19:58.245641 1299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:19:58.229999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55194 (local address 127.16.217.62:34019)
0620 20:19:58.230802 (+ 803us) server_negotiation.cc:206] Beginning negotiation
0620 20:19:58.230808 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:19:58.230826 (+ 18us) server_negotiation.cc:407] Connection header received
0620 20:19:58.230880 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:19:58.230886 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:19:58.230959 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:19:58.231068 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:19:58.232732 (+ 1664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:19:58.233736 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:19:58.235169 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:19:58.235847 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:19:58.235985 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:19:58.239074 (+ 3089us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:19:58.239081 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:19:58.239084 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:19:58.239168 (+ 84us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:19:58.242779 (+ 3611us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:19:58.243606 (+ 827us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:19:58.243611 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:19:58.243614 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:19:58.243731 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:19:58.244165 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:19:58.244170 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:19:58.244173 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:19:58.244507 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:19:58.244771 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0620 20:19:58.245027 (+ 256us) server_negotiation.cc:299] Negotiation successful
0620 20:19:58.245245 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":680,"thread_start_us":556,"threads_started":1}
I20260620 20:19:58.246874 1295 heartbeater.cc:347] Connected to a master server at 127.16.217.62:34019
I20260620 20:19:58.247236 1295 heartbeater.cc:464] Registering TS with master...
I20260620 20:19:58.248004 1295 heartbeater.cc:511] Master 127.16.217.62:34019 requested a full tablet report, sending...
I20260620 20:19:58.249362 846 ts_manager.cc:194] Registered new tserver with Master: 24811d2b3eec4754a9194eabaa73b41f (127.16.217.3:46007)
I20260620 20:19:58.250244 846 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.217.3@KRBTEST.COM'} at 127.16.217.3:46563
I20260620 20:19:58.260325 17252 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 20:19:58.266974 1299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:19:58.261187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55198 (local address 127.16.217.62:34019)
0620 20:19:58.261426 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0620 20:19:58.261435 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 20:19:58.261818 (+ 383us) server_negotiation.cc:407] Connection header received
0620 20:19:58.262048 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:19:58.262052 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:19:58.262180 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:19:58.262296 (+ 116us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 20:19:58.262680 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:19:58.263422 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:19:58.264924 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:19:58.265373 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:19:58.265467 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:19:58.266215 (+ 748us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 20:19:58.266441 (+ 226us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0620 20:19:58.266737 (+ 296us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:55198: token expired
Metrics: {"server-negotiator.queue_time_us":58}
W20260620 20:19:58.267129 1299 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:55198: token expired
W20260620 20:19:58.267719 767 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.16.217.62:34019)
I20260620 20:19:58.281126 1299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:19:58.268175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55206 (local address 127.16.217.62:34019)
0620 20:19:58.268350 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0620 20:19:58.268358 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 20:19:58.268380 (+ 22us) server_negotiation.cc:407] Connection header received
0620 20:19:58.268620 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:19:58.268625 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:19:58.268687 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:19:58.268791 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:19:58.270169 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:19:58.270912 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:19:58.272284 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:19:58.272728 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:19:58.272862 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:19:58.275894 (+ 3032us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:19:58.275900 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:19:58.275906 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:19:58.275944 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:19:58.278437 (+ 2493us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:19:58.279131 (+ 694us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:19:58.279144 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:19:58.279148 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:19:58.279283 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:19:58.279656 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:19:58.279661 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:19:58.279665 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:19:58.279990 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:19:58.280252 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0620 20:19:58.280548 (+ 296us) server_negotiation.cc:299] Negotiation successful
0620 20:19:58.280788 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 20:19:58.284722 846 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55206:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 20:19:58.292752 846 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 20:19:58.316779 1308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:19:58.306657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58060 (local address 127.16.217.3:46007)
0620 20:19:58.307784 (+ 1127us) server_negotiation.cc:206] Beginning negotiation
0620 20:19:58.307793 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 20:19:58.307823 (+ 30us) server_negotiation.cc:407] Connection header received
0620 20:19:58.307922 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:19:58.307928 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:19:58.308166 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:19:58.308526 (+ 360us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:19:58.309703 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:19:58.311349 (+ 1646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:19:58.313957 (+ 2608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:19:58.315424 (+ 1467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:19:58.315554 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:19:58.315817 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0620 20:19:58.316150 (+ 333us) server_negotiation.cc:299] Negotiation successful
0620 20:19:58.316322 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":956,"thread_start_us":793,"threads_started":1}
I20260620 20:19:58.316825 1307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:19:58.306205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.3:46007 (local address 127.0.0.1:58060)
0620 20:19:58.306978 (+ 773us) negotiation.cc:107] Waiting for socket to connect
0620 20:19:58.306993 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 20:19:58.307148 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:19:58.308396 (+ 1248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:19:58.308400 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:19:58.308427 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:19:58.309455 (+ 1028us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:19:58.309478 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:19:58.311495 (+ 2017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:19:58.311501 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:19:58.313713 (+ 2212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:19:58.313728 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:19:58.315668 (+ 1940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:19:58.315675 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:19:58.315884 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:19:58.315941 (+ 57us) client_negotiation.cc:769] Sending connection context
0620 20:19:58.316068 (+ 127us) client_negotiation.cc:240] Negotiation successful
0620 20:19:58.316334 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":649,"mutex_wait_us":155,"spinlock_wait_cycles":4864,"thread_start_us":1460,"threads_started":1}
I20260620 20:19:58.317058 1305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:19:58.305229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34052 (local address 127.16.217.1:38751)
0620 20:19:58.305995 (+ 766us) server_negotiation.cc:206] Beginning negotiation
0620 20:19:58.306006 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 20:19:58.306976 (+ 970us) server_negotiation.cc:407] Connection header received
0620 20:19:58.307416 (+ 440us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:19:58.307429 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:19:58.307742 (+ 313us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:19:58.307920 (+ 178us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:19:58.310226 (+ 2306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:19:58.311713 (+ 1487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:19:58.314450 (+ 2737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:19:58.315634 (+ 1184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:19:58.315743 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:19:58.315971 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0620 20:19:58.316580 (+ 609us) server_negotiation.cc:299] Negotiation successful
0620 20:19:58.316746 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":573,"thread_start_us":416,"threads_started":1}
I20260620 20:19:58.317611 1309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:19:58.307489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.2:33863 (local address 127.0.0.1:60062)
0620 20:19:58.309635 (+ 2146us) negotiation.cc:107] Waiting for socket to connect
0620 20:19:58.309655 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 20:19:58.309767 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:19:58.310388 (+ 621us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:19:58.310393 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:19:58.310417 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:19:58.310761 (+ 344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:19:58.310770 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:19:58.312373 (+ 1603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:19:58.312380 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:19:58.314666 (+ 2286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:19:58.314680 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:19:58.316788 (+ 2108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:19:58.316796 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:19:58.317010 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:19:58.317039 (+ 29us) client_negotiation.cc:769] Sending connection context
0620 20:19:58.317123 (+ 84us) client_negotiation.cc:240] Negotiation successful
0620 20:19:58.317241 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1988,"thread_start_us":1817,"threads_started":1}
I20260620 20:19:58.317901 1306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:19:58.305177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.1:38751 (local address 127.0.0.1:34052)
0620 20:19:58.306820 (+ 1643us) negotiation.cc:107] Waiting for socket to connect
0620 20:19:58.306868 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0620 20:19:58.307148 (+ 280us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:19:58.307980 (+ 832us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:19:58.307989 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:19:58.308031 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:19:58.310038 (+ 2007us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:19:58.310052 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:19:58.311845 (+ 1793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:19:58.311851 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:19:58.314274 (+ 2423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:19:58.314294 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:19:58.316201 (+ 1907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:19:58.316207 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:19:58.316363 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:19:58.316391 (+ 28us) client_negotiation.cc:769] Sending connection context
0620 20:19:58.317361 (+ 970us) client_negotiation.cc:240] Negotiation successful
0620 20:19:58.317454 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1310,"mutex_wait_us":40,"spinlock_wait_cycles":6784,"thread_start_us":1106,"threads_started":1}
I20260620 20:19:58.319454 1310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:19:58.308006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60062 (local address 127.16.217.2:33863)
0620 20:19:58.308572 (+ 566us) server_negotiation.cc:206] Beginning negotiation
0620 20:19:58.308580 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 20:19:58.309752 (+ 1172us) server_negotiation.cc:407] Connection header received
0620 20:19:58.309955 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:19:58.309965 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:19:58.310201 (+ 236us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:19:58.310407 (+ 206us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:19:58.310907 (+ 500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:19:58.312214 (+ 1307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:19:58.314820 (+ 2606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:19:58.316337 (+ 1517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:19:58.316476 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:19:58.318903 (+ 2427us) server_negotiation.cc:1035] Waiting for connection context
0620 20:19:58.318981 (+ 78us) server_negotiation.cc:299] Negotiation successful
0620 20:19:58.319143 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":410,"thread_start_us":273,"threads_started":1}
I20260620 20:19:58.321343 1230 tablet_service.cc:1467] Processing CreateTablet for tablet c82c3f55f5654f5aa0d28a92481d18de (DEFAULT_TABLE table=test-table [id=3da3f39112314781a7c98a2f8bf87316]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:19:58.322211 1099 tablet_service.cc:1467] Processing CreateTablet for tablet c82c3f55f5654f5aa0d28a92481d18de (DEFAULT_TABLE table=test-table [id=3da3f39112314781a7c98a2f8bf87316]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:19:58.322773 1230 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c82c3f55f5654f5aa0d28a92481d18de. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:19:58.323304 1099 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c82c3f55f5654f5aa0d28a92481d18de. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:19:58.325713 967 tablet_service.cc:1467] Processing CreateTablet for tablet c82c3f55f5654f5aa0d28a92481d18de (DEFAULT_TABLE table=test-table [id=3da3f39112314781a7c98a2f8bf87316]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:19:58.327977 967 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c82c3f55f5654f5aa0d28a92481d18de. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:19:58.329329 1032 heartbeater.cc:503] Master 127.16.217.62:34019 was elected leader, sending a full tablet report...
I20260620 20:19:58.336357 1311 tablet_bootstrap.cc:492] T c82c3f55f5654f5aa0d28a92481d18de P bb9261ea26ae46a580415794f87a5ee0: Bootstrap starting.
I20260620 20:19:58.338570 1312 tablet_bootstrap.cc:492] T c82c3f55f5654f5aa0d28a92481d18de P 24811d2b3eec4754a9194eabaa73b41f: Bootstrap starting.
I20260620 20:19:58.340209 1311 tablet_bootstrap.cc:654] T c82c3f55f5654f5aa0d28a92481d18de P bb9261ea26ae46a580415794f87a5ee0: Neither blocks nor log segments found. Creating new log.
I20260620 20:19:58.341471 1311 log.cc:824] T c82c3f55f5654f5aa0d28a92481d18de P bb9261ea26ae46a580415794f87a5ee0: Log is configured to *not* fsync() on all Append() calls
I20260620 20:19:58.342016 1312 tablet_bootstrap.cc:654] T c82c3f55f5654f5aa0d28a92481d18de P 24811d2b3eec4754a9194eabaa73b41f: Neither blocks nor log segments found. Creating new log.
I20260620 20:19:58.343248 1312 log.cc:824] T c82c3f55f5654f5aa0d28a92481d18de P 24811d2b3eec4754a9194eabaa73b41f: Log is configured to *not* fsync() on all Append() calls
I20260620 20:19:58.343884 1314 tablet_bootstrap.cc:492] T c82c3f55f5654f5aa0d28a92481d18de P 48ddf12e1ab742819abba58564f9d6b5: Bootstrap starting.
I20260620 20:19:58.344480 1311 tablet_bootstrap.cc:492] T c82c3f55f5654f5aa0d28a92481d18de P bb9261ea26ae46a580415794f87a5ee0: No bootstrap required, opened a new log
I20260620 20:19:58.344830 1311 ts_tablet_manager.cc:1397] T c82c3f55f5654f5aa0d28a92481d18de P bb9261ea26ae46a580415794f87a5ee0: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.004s
I20260620 20:19:58.346519 1312 tablet_bootstrap.cc:492] T c82c3f55f5654f5aa0d28a92481d18de P 24811d2b3eec4754a9194eabaa73b41f: No bootstrap required, opened a new log
I20260620 20:19:58.346797 1312 ts_tablet_manager.cc:1397] T c82c3f55f5654f5aa0d28a92481d18de P 24811d2b3eec4754a9194eabaa73b41f: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260620 20:19:58.347101 1314 tablet_bootstrap.cc:654] T c82c3f55f5654f5aa0d28a92481d18de P 48ddf12e1ab742819abba58564f9d6b5: Neither blocks nor log segments found. Creating new log.
I20260620 20:19:58.348276 1314 log.cc:824] T c82c3f55f5654f5aa0d28a92481d18de P 48ddf12e1ab742819abba58564f9d6b5: Log is configured to *not* fsync() on all Append() calls
I20260620 20:19:58.354605 1314 tablet_bootstrap.cc:492] T c82c3f55f5654f5aa0d28a92481d18de P 48ddf12e1ab742819abba58564f9d6b5: No bootstrap required, opened a new log
I20260620 20:19:58.354885 1314 ts_tablet_manager.cc:1397] T c82c3f55f5654f5aa0d28a92481d18de P 48ddf12e1ab742819abba58564f9d6b5: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.004s
I20260620 20:19:58.355695 1311 raft_consensus.cc:348] T c82c3f55f5654f5aa0d28a92481d18de P bb9261ea26ae46a580415794f87a5ee0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48ddf12e1ab742819abba58564f9d6b5" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 38751 } } peers { permanent_uuid: "24811d2b3eec4754a9194eabaa73b41f" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 46007 } } peers { permanent_uuid: "bb9261ea26ae46a580415794f87a5ee0" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 33863 } }
I20260620 20:19:58.356184 1311 raft_consensus.cc:374] T c82c3f55f5654f5aa0d28a92481d18de P bb9261ea26ae46a580415794f87a5ee0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:19:58.356287 1311 raft_consensus.cc:729] T c82c3f55f5654f5aa0d28a92481d18de P bb9261ea26ae46a580415794f87a5ee0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb9261ea26ae46a580415794f87a5ee0, State: Initialized, Role: FOLLOWER
I20260620 20:19:58.356920 1311 consensus_queue.cc:260] T c82c3f55f5654f5aa0d28a92481d18de P bb9261ea26ae46a580415794f87a5ee0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48ddf12e1ab742819abba58564f9d6b5" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 38751 } } peers { permanent_uuid: "24811d2b3eec4754a9194eabaa73b41f" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 46007 } } peers { permanent_uuid: "bb9261ea26ae46a580415794f87a5ee0" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 33863 } }
I20260620 20:19:58.357427 1312 raft_consensus.cc:348] T c82c3f55f5654f5aa0d28a92481d18de P 24811d2b3eec4754a9194eabaa73b41f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48ddf12e1ab742819abba58564f9d6b5" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 38751 } } peers { permanent_uuid: "24811d2b3eec4754a9194eabaa73b41f" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 46007 } } peers { permanent_uuid: "bb9261ea26ae46a580415794f87a5ee0" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 33863 } }
I20260620 20:19:58.357966 1312 raft_consensus.cc:374] T c82c3f55f5654f5aa0d28a92481d18de P 24811d2b3eec4754a9194eabaa73b41f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:19:58.358109 1312 raft_consensus.cc:729] T c82c3f55f5654f5aa0d28a92481d18de P 24811d2b3eec4754a9194eabaa73b41f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 24811d2b3eec4754a9194eabaa73b41f, State: Initialized, Role: FOLLOWER
I20260620 20:19:58.358127 1164 heartbeater.cc:503] Master 127.16.217.62:34019 was elected leader, sending a full tablet report...
I20260620 20:19:58.358752 1311 ts_tablet_manager.cc:1428] T c82c3f55f5654f5aa0d28a92481d18de P bb9261ea26ae46a580415794f87a5ee0: Time spent starting tablet: real 0.014s user 0.014s sys 0.001s
I20260620 20:19:58.358826 1312 consensus_queue.cc:260] T c82c3f55f5654f5aa0d28a92481d18de P 24811d2b3eec4754a9194eabaa73b41f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48ddf12e1ab742819abba58564f9d6b5" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 38751 } } peers { permanent_uuid: "24811d2b3eec4754a9194eabaa73b41f" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 46007 } } peers { permanent_uuid: "bb9261ea26ae46a580415794f87a5ee0" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 33863 } }
I20260620 20:19:58.361446 1295 heartbeater.cc:503] Master 127.16.217.62:34019 was elected leader, sending a full tablet report...
I20260620 20:19:58.362161 1312 ts_tablet_manager.cc:1428] T c82c3f55f5654f5aa0d28a92481d18de P 24811d2b3eec4754a9194eabaa73b41f: Time spent starting tablet: real 0.015s user 0.016s sys 0.000s
I20260620 20:19:58.364476 1314 raft_consensus.cc:348] T c82c3f55f5654f5aa0d28a92481d18de P 48ddf12e1ab742819abba58564f9d6b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48ddf12e1ab742819abba58564f9d6b5" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 38751 } } peers { permanent_uuid: "24811d2b3eec4754a9194eabaa73b41f" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 46007 } } peers { permanent_uuid: "bb9261ea26ae46a580415794f87a5ee0" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 33863 } }
I20260620 20:19:58.364857 1314 raft_consensus.cc:374] T c82c3f55f5654f5aa0d28a92481d18de P 48ddf12e1ab742819abba58564f9d6b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:19:58.364979 1314 raft_consensus.cc:729] T c82c3f55f5654f5aa0d28a92481d18de P 48ddf12e1ab742819abba58564f9d6b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48ddf12e1ab742819abba58564f9d6b5, State: Initialized, Role: FOLLOWER
I20260620 20:19:58.365518 1314 consensus_queue.cc:260] T c82c3f55f5654f5aa0d28a92481d18de P 48ddf12e1ab742819abba58564f9d6b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48ddf12e1ab742819abba58564f9d6b5" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 38751 } } peers { permanent_uuid: "24811d2b3eec4754a9194eabaa73b41f" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 46007 } } peers { permanent_uuid: "bb9261ea26ae46a580415794f87a5ee0" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 33863 } }
I20260620 20:19:58.367331 1314 ts_tablet_manager.cc:1428] T c82c3f55f5654f5aa0d28a92481d18de P 48ddf12e1ab742819abba58564f9d6b5: Time spent starting tablet: real 0.012s user 0.009s sys 0.004s
W20260620 20:19:58.472437 1296 tablet.cc:2369] T c82c3f55f5654f5aa0d28a92481d18de P 24811d2b3eec4754a9194eabaa73b41f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 20:19:58.503417 1165 tablet.cc:2369] T c82c3f55f5654f5aa0d28a92481d18de P bb9261ea26ae46a580415794f87a5ee0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 20:19:58.512223 1319 raft_consensus.cc:482] T c82c3f55f5654f5aa0d28a92481d18de P 48ddf12e1ab742819abba58564f9d6b5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 20:19:58.512545 1319 raft_consensus.cc:504] T c82c3f55f5654f5aa0d28a92481d18de P 48ddf12e1ab742819abba58564f9d6b5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48ddf12e1ab742819abba58564f9d6b5" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 38751 } } peers { permanent_uuid: "24811d2b3eec4754a9194eabaa73b41f" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 46007 } } peers { permanent_uuid: "bb9261ea26ae46a580415794f87a5ee0" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 33863 } }
I20260620 20:19:58.514403 1319 leader_election.cc:288] T c82c3f55f5654f5aa0d28a92481d18de P 48ddf12e1ab742819abba58564f9d6b5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 24811d2b3eec4754a9194eabaa73b41f (127.16.217.3:46007), bb9261ea26ae46a580415794f87a5ee0 (127.16.217.2:33863)
I20260620 20:19:58.521440 1308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:19:58.514864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.217.1:42967 (local address 127.16.217.3:46007)
0620 20:19:58.515108 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0620 20:19:58.515117 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 20:19:58.515824 (+ 707us) server_negotiation.cc:407] Connection header received
0620 20:19:58.516082 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:19:58.516089 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:19:58.516159 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:19:58.516266 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:19:58.516963 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:19:58.517702 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:19:58.519624 (+ 1922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:19:58.520404 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:19:58.520572 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:19:58.520664 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0620 20:19:58.521024 (+ 360us) server_negotiation.cc:299] Negotiation successful
0620 20:19:58.521158 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260620 20:19:58.521479 1310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:19:58.515039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.217.1:53743 (local address 127.16.217.2:33863)
0620 20:19:58.515276 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0620 20:19:58.515284 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 20:19:58.515949 (+ 665us) server_negotiation.cc:407] Connection header received
0620 20:19:58.516103 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:19:58.516107 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:19:58.516169 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:19:58.516270 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:19:58.517019 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:19:58.517741 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:19:58.519677 (+ 1936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:19:58.520422 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:19:58.520611 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:19:58.520682 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 20:19:58.521087 (+ 405us) server_negotiation.cc:299] Negotiation successful
0620 20:19:58.521224 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260620 20:19:58.521588 1320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:19:58.514735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.3:46007 (local address 127.16.217.1:42967)
0620 20:19:58.515680 (+ 945us) negotiation.cc:107] Waiting for socket to connect
0620 20:19:58.515699 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 20:19:58.515886 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:19:58.516281 (+ 395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:19:58.516285 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:19:58.516308 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:19:58.516805 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:19:58.516819 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:19:58.517842 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:19:58.517850 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:19:58.519476 (+ 1626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:19:58.519487 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:19:58.520641 (+ 1154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:19:58.520649 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:19:58.520858 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:19:58.520897 (+ 39us) client_negotiation.cc:769] Sending connection context
0620 20:19:58.520970 (+ 73us) client_negotiation.cc:240] Negotiation successful
0620 20:19:58.521105 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":802,"mutex_wait_us":23,"spinlock_wait_cycles":41856,"thread_start_us":604,"threads_started":1}
I20260620 20:19:58.521658 1321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:19:58.514876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.2:33863 (local address 127.16.217.1:53743)
0620 20:19:58.515827 (+ 951us) negotiation.cc:107] Waiting for socket to connect
0620 20:19:58.515846 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 20:19:58.515988 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:19:58.516287 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:19:58.516291 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:19:58.516382 (+ 91us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:19:58.516832 (+ 450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:19:58.516841 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:19:58.518053 (+ 1212us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:19:58.518058 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:19:58.519493 (+ 1435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:19:58.519504 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:19:58.520678 (+ 1174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:19:58.520686 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:19:58.520905 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:19:58.520927 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 20:19:58.521040 (+ 113us) client_negotiation.cc:240] Negotiation successful
0620 20:19:58.521156 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":806,"mutex_wait_us":56,"spinlock_wait_cycles":25856,"thread_start_us":610,"threads_started":1}
I20260620 20:19:58.522763 1119 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c82c3f55f5654f5aa0d28a92481d18de" candidate_uuid: "48ddf12e1ab742819abba58564f9d6b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb9261ea26ae46a580415794f87a5ee0" is_pre_election: true
I20260620 20:19:58.522766 1250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c82c3f55f5654f5aa0d28a92481d18de" candidate_uuid: "48ddf12e1ab742819abba58564f9d6b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "24811d2b3eec4754a9194eabaa73b41f" is_pre_election: true
I20260620 20:19:58.523236 1250 raft_consensus.cc:2454] T c82c3f55f5654f5aa0d28a92481d18de P 24811d2b3eec4754a9194eabaa73b41f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48ddf12e1ab742819abba58564f9d6b5 in term 0.
I20260620 20:19:58.523234 1119 raft_consensus.cc:2454] T c82c3f55f5654f5aa0d28a92481d18de P bb9261ea26ae46a580415794f87a5ee0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48ddf12e1ab742819abba58564f9d6b5 in term 0.
I20260620 20:19:58.523934 922 leader_election.cc:302] T c82c3f55f5654f5aa0d28a92481d18de P 48ddf12e1ab742819abba58564f9d6b5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 24811d2b3eec4754a9194eabaa73b41f, 48ddf12e1ab742819abba58564f9d6b5, bb9261ea26ae46a580415794f87a5ee0; no voters:
I20260620 20:19:58.524358 1319 raft_consensus.cc:2781] T c82c3f55f5654f5aa0d28a92481d18de P 48ddf12e1ab742819abba58564f9d6b5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 20:19:58.524503 1319 raft_consensus.cc:482] T c82c3f55f5654f5aa0d28a92481d18de P 48ddf12e1ab742819abba58564f9d6b5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 20:19:58.524605 1319 raft_consensus.cc:3037] T c82c3f55f5654f5aa0d28a92481d18de P 48ddf12e1ab742819abba58564f9d6b5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:19:58.526954 1319 raft_consensus.cc:504] T c82c3f55f5654f5aa0d28a92481d18de P 48ddf12e1ab742819abba58564f9d6b5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48ddf12e1ab742819abba58564f9d6b5" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 38751 } } peers { permanent_uuid: "24811d2b3eec4754a9194eabaa73b41f" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 46007 } } peers { permanent_uuid: "bb9261ea26ae46a580415794f87a5ee0" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 33863 } }
I20260620 20:19:58.527668 1319 leader_election.cc:288] T c82c3f55f5654f5aa0d28a92481d18de P 48ddf12e1ab742819abba58564f9d6b5 [CANDIDATE]: Term 1 election: Requested vote from peers 24811d2b3eec4754a9194eabaa73b41f (127.16.217.3:46007), bb9261ea26ae46a580415794f87a5ee0 (127.16.217.2:33863)
I20260620 20:19:58.528103 1250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c82c3f55f5654f5aa0d28a92481d18de" candidate_uuid: "48ddf12e1ab742819abba58564f9d6b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "24811d2b3eec4754a9194eabaa73b41f"
I20260620 20:19:58.528129 1119 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c82c3f55f5654f5aa0d28a92481d18de" candidate_uuid: "48ddf12e1ab742819abba58564f9d6b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb9261ea26ae46a580415794f87a5ee0"
I20260620 20:19:58.528321 1250 raft_consensus.cc:3037] T c82c3f55f5654f5aa0d28a92481d18de P 24811d2b3eec4754a9194eabaa73b41f [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:19:58.528316 1119 raft_consensus.cc:3037] T c82c3f55f5654f5aa0d28a92481d18de P bb9261ea26ae46a580415794f87a5ee0 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:19:58.530536 1250 raft_consensus.cc:2454] T c82c3f55f5654f5aa0d28a92481d18de P 24811d2b3eec4754a9194eabaa73b41f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48ddf12e1ab742819abba58564f9d6b5 in term 1.
I20260620 20:19:58.530568 1119 raft_consensus.cc:2454] T c82c3f55f5654f5aa0d28a92481d18de P bb9261ea26ae46a580415794f87a5ee0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48ddf12e1ab742819abba58564f9d6b5 in term 1.
I20260620 20:19:58.530951 921 leader_election.cc:302] T c82c3f55f5654f5aa0d28a92481d18de P 48ddf12e1ab742819abba58564f9d6b5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 24811d2b3eec4754a9194eabaa73b41f, 48ddf12e1ab742819abba58564f9d6b5; no voters:
I20260620 20:19:58.531293 1319 raft_consensus.cc:2781] T c82c3f55f5654f5aa0d28a92481d18de P 48ddf12e1ab742819abba58564f9d6b5 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:19:58.531728 1319 raft_consensus.cc:686] T c82c3f55f5654f5aa0d28a92481d18de P 48ddf12e1ab742819abba58564f9d6b5 [term 1 LEADER]: Becoming Leader. State: Replica: 48ddf12e1ab742819abba58564f9d6b5, State: Running, Role: LEADER
I20260620 20:19:58.532269 1319 consensus_queue.cc:237] T c82c3f55f5654f5aa0d28a92481d18de P 48ddf12e1ab742819abba58564f9d6b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48ddf12e1ab742819abba58564f9d6b5" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 38751 } } peers { permanent_uuid: "24811d2b3eec4754a9194eabaa73b41f" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 46007 } } peers { permanent_uuid: "bb9261ea26ae46a580415794f87a5ee0" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 33863 } }
I20260620 20:19:58.536790 845 catalog_manager.cc:5495] T c82c3f55f5654f5aa0d28a92481d18de P 48ddf12e1ab742819abba58564f9d6b5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 48ddf12e1ab742819abba58564f9d6b5 (127.16.217.1). New cstate: current_term: 1 leader_uuid: "48ddf12e1ab742819abba58564f9d6b5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48ddf12e1ab742819abba58564f9d6b5" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 38751 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "24811d2b3eec4754a9194eabaa73b41f" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 46007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bb9261ea26ae46a580415794f87a5ee0" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 33863 } health_report { overall_health: UNKNOWN } } }
W20260620 20:19:58.542147 1033 tablet.cc:2369] T c82c3f55f5654f5aa0d28a92481d18de P 48ddf12e1ab742819abba58564f9d6b5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 20:19:58.559551 1305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:19:58.554182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34068 (local address 127.16.217.1:38751)
0620 20:19:58.554402 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0620 20:19:58.554407 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:19:58.554424 (+ 17us) server_negotiation.cc:407] Connection header received
0620 20:19:58.554612 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:19:58.554616 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:19:58.554730 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:19:58.554846 (+ 116us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 20:19:58.555203 (+ 357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:19:58.555944 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:19:58.557274 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:19:58.557670 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:19:58.557759 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:19:58.558396 (+ 637us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 20:19:58.558931 (+ 535us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 20:19:58.559036 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0620 20:19:58.559196 (+ 160us) server_negotiation.cc:299] Negotiation successful
0620 20:19:58.559257 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
I20260620 20:19:58.568022 1119 raft_consensus.cc:1261] T c82c3f55f5654f5aa0d28a92481d18de P bb9261ea26ae46a580415794f87a5ee0 [term 1 FOLLOWER]: Refusing update from remote peer 48ddf12e1ab742819abba58564f9d6b5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:19:58.568032 1250 raft_consensus.cc:1261] T c82c3f55f5654f5aa0d28a92481d18de P 24811d2b3eec4754a9194eabaa73b41f [term 1 FOLLOWER]: Refusing update from remote peer 48ddf12e1ab742819abba58564f9d6b5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:19:58.568992 1322 consensus_queue.cc:1035] T c82c3f55f5654f5aa0d28a92481d18de P 48ddf12e1ab742819abba58564f9d6b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "24811d2b3eec4754a9194eabaa73b41f" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 46007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:19:58.569319 1319 consensus_queue.cc:1035] T c82c3f55f5654f5aa0d28a92481d18de P 48ddf12e1ab742819abba58564f9d6b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb9261ea26ae46a580415794f87a5ee0" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 33863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:19:58.581718 1325 mvcc.cc:205] Tried to move back new op lower bound from 7299017926924832768 to 7299017926792093696. Current Snapshot: MvccSnapshot[applied={T|T < 7299017926924832768}]
I20260620 20:19:58.596203 845 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55206:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 20:19:58.596534 845 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55206:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 20:19:58.600474 845 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8e39d1abbd4e46a7bc73a63bfb83dfd0: Sending DeleteTablet for 3 replicas of tablet c82c3f55f5654f5aa0d28a92481d18de
I20260620 20:19:58.601760 1230 tablet_service.cc:1514] Processing DeleteTablet for tablet c82c3f55f5654f5aa0d28a92481d18de with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:19:58 UTC) from {username='kudu', principal='kudu/127.16.217.62@KRBTEST.COM'} at 127.0.0.1:58060
I20260620 20:19:58.602104 1099 tablet_service.cc:1514] Processing DeleteTablet for tablet c82c3f55f5654f5aa0d28a92481d18de with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:19:58 UTC) from {username='kudu', principal='kudu/127.16.217.62@KRBTEST.COM'} at 127.0.0.1:60062
I20260620 20:19:58.602196 967 tablet_service.cc:1514] Processing DeleteTablet for tablet c82c3f55f5654f5aa0d28a92481d18de with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:19:58 UTC) from {username='kudu', principal='kudu/127.16.217.62@KRBTEST.COM'} at 127.0.0.1:34052
I20260620 20:19:58.603296 17252 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu pid 883
I20260620 20:19:58.642915 1335 tablet_replica.cc:330] T c82c3f55f5654f5aa0d28a92481d18de P 24811d2b3eec4754a9194eabaa73b41f: stopping tablet replica
I20260620 20:19:58.641860 1305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:19:58.605399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34082 (local address 127.16.217.1:38751)
0620 20:19:58.605636 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0620 20:19:58.605646 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 20:19:58.605671 (+ 25us) server_negotiation.cc:407] Connection header received
0620 20:19:58.605752 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:19:58.605761 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:19:58.605898 (+ 137us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:19:58.606043 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:19:58.609276 (+ 3233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:19:58.610998 (+ 1722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:19:58.613767 (+ 2769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:19:58.614809 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:19:58.614979 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:19:58.627244 (+ 12265us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:19:58.627271 (+ 27us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:19:58.627300 (+ 29us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:19:58.627379 (+ 79us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:19:58.633428 (+ 6049us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:19:58.634959 (+ 1531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:19:58.634976 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:19:58.634982 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:19:58.635272 (+ 290us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:19:58.637164 (+ 1892us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:19:58.637172 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:19:58.637179 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:19:58.637843 (+ 664us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:19:58.638413 (+ 570us) server_negotiation.cc:1035] Waiting for connection context
0620 20:19:58.640811 (+ 2398us) server_negotiation.cc:299] Negotiation successful
0620 20:19:58.641211 (+ 400us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260620 20:19:58.643057 1337 tablet_replica.cc:330] T c82c3f55f5654f5aa0d28a92481d18de P 48ddf12e1ab742819abba58564f9d6b5: stopping tablet replica
I20260620 20:19:58.643859 1337 raft_consensus.cc:2229] T c82c3f55f5654f5aa0d28a92481d18de P 48ddf12e1ab742819abba58564f9d6b5 [term 1 LEADER]: Raft consensus shutting down.
I20260620 20:19:58.644419 1027 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:19:58.644683 1335 raft_consensus.cc:2229] T c82c3f55f5654f5aa0d28a92481d18de P 24811d2b3eec4754a9194eabaa73b41f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 20:19:58.645041 1336 tablet_replica.cc:330] T c82c3f55f5654f5aa0d28a92481d18de P bb9261ea26ae46a580415794f87a5ee0: stopping tablet replica
I20260620 20:19:58.645493 1335 raft_consensus.cc:2258] T c82c3f55f5654f5aa0d28a92481d18de P 24811d2b3eec4754a9194eabaa73b41f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:19:58.645591 1336 raft_consensus.cc:2229] T c82c3f55f5654f5aa0d28a92481d18de P bb9261ea26ae46a580415794f87a5ee0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 20:19:58.646354 1336 raft_consensus.cc:2258] T c82c3f55f5654f5aa0d28a92481d18de P bb9261ea26ae46a580415794f87a5ee0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:19:58.648115 1336 ts_tablet_manager.cc:1905] T c82c3f55f5654f5aa0d28a92481d18de P bb9261ea26ae46a580415794f87a5ee0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:19:58.648348 1335 ts_tablet_manager.cc:1905] T c82c3f55f5654f5aa0d28a92481d18de P 24811d2b3eec4754a9194eabaa73b41f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:19:58.653784 1336 ts_tablet_manager.cc:1918] T c82c3f55f5654f5aa0d28a92481d18de P bb9261ea26ae46a580415794f87a5ee0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 20:19:58.653972 1336 log.cc:1196] T c82c3f55f5654f5aa0d28a92481d18de P bb9261ea26ae46a580415794f87a5ee0: Deleting WAL directory at /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986536687152-17252-0/minicluster-data/ts-1/wal/wals/c82c3f55f5654f5aa0d28a92481d18de
I20260620 20:19:58.654456 1336 ts_tablet_manager.cc:1939] T c82c3f55f5654f5aa0d28a92481d18de P bb9261ea26ae46a580415794f87a5ee0: Deleting consensus metadata
I20260620 20:19:58.656088 832 catalog_manager.cc:4841] TS bb9261ea26ae46a580415794f87a5ee0 (127.16.217.2:33863): tablet c82c3f55f5654f5aa0d28a92481d18de (table test-table [id=3da3f39112314781a7c98a2f8bf87316]) successfully deleted
I20260620 20:19:58.658213 1335 ts_tablet_manager.cc:1918] T c82c3f55f5654f5aa0d28a92481d18de P 24811d2b3eec4754a9194eabaa73b41f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 20:19:58.658380 1335 log.cc:1196] T c82c3f55f5654f5aa0d28a92481d18de P 24811d2b3eec4754a9194eabaa73b41f: Deleting WAL directory at /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986536687152-17252-0/minicluster-data/ts-2/wal/wals/c82c3f55f5654f5aa0d28a92481d18de
I20260620 20:19:58.658860 1335 ts_tablet_manager.cc:1939] T c82c3f55f5654f5aa0d28a92481d18de P 24811d2b3eec4754a9194eabaa73b41f: Deleting consensus metadata
I20260620 20:19:58.660425 831 catalog_manager.cc:4841] TS 24811d2b3eec4754a9194eabaa73b41f (127.16.217.3:46007): tablet c82c3f55f5654f5aa0d28a92481d18de (table test-table [id=3da3f39112314781a7c98a2f8bf87316]) successfully deleted
I20260620 20:19:58.754194 1337 raft_consensus.cc:2258] T c82c3f55f5654f5aa0d28a92481d18de P 48ddf12e1ab742819abba58564f9d6b5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:19:58.754781 17252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu with pid 883
W20260620 20:19:58.779914 832 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.16.217.1:38751 (error 108)
I20260620 20:19:58.780185 17252 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu pid 1036
I20260620 20:19:58.782037 1306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:19:58.781552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.1:38751 (local address 127.0.0.1:34088)
0620 20:19:58.781727 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0620 20:19:58.781890 (+ 163us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.217.1:38751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260620 20:19:58.782323 832 catalog_manager.cc:4584] TS 48ddf12e1ab742819abba58564f9d6b5 (127.16.217.1:38751): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c82c3f55f5654f5aa0d28a92481d18de: Network error: Client connection negotiation failed: client connection to 127.16.217.1:38751: connect: Connection refused (error 111)
I20260620 20:19:58.795806 1310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:19:58.781792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60076 (local address 127.16.217.2:33863)
0620 20:19:58.782025 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0620 20:19:58.782031 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:19:58.782051 (+ 20us) server_negotiation.cc:407] Connection header received
0620 20:19:58.782109 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:19:58.782115 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:19:58.782192 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:19:58.782306 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:19:58.783687 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:19:58.784809 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:19:58.786738 (+ 1929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:19:58.787371 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:19:58.787503 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:19:58.790248 (+ 2745us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:19:58.790259 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:19:58.790275 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:19:58.790317 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:19:58.793383 (+ 3066us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:19:58.794179 (+ 796us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:19:58.794187 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:19:58.794189 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:19:58.794312 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:19:58.794602 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:19:58.794605 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:19:58.794607 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:19:58.794958 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:19:58.795178 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0620 20:19:58.795383 (+ 205us) server_negotiation.cc:299] Negotiation successful
0620 20:19:58.795544 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102}
I20260620 20:19:58.796427 1159 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:19:58.836233 1306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:19:58.835715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.1:38751 (local address 127.0.0.1:34100)
0620 20:19:58.835952 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0620 20:19:58.836077 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.217.1:38751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
I20260620 20:19:58.837314 1306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:19:58.836954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.1:38751 (local address 127.0.0.1:34116)
0620 20:19:58.837113 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0620 20:19:58.837204 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.217.1:38751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260620 20:19:58.886933 17252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu with pid 1036
I20260620 20:19:58.897924 1306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:19:58.897300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.1:38751 (local address 127.0.0.1:34118)
0620 20:19:58.897551 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0620 20:19:58.897679 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.217.1:38751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260620 20:19:58.899246 1306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:19:58.898768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.1:38751 (local address 127.0.0.1:34128)
0620 20:19:58.898983 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0620 20:19:58.899079 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.217.1:38751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260620 20:19:58.904628 17252 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu pid 1167
I20260620 20:19:58.918177 1308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:19:58.905423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58062 (local address 127.16.217.3:46007)
0620 20:19:58.905611 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0620 20:19:58.905616 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:19:58.905636 (+ 20us) server_negotiation.cc:407] Connection header received
0620 20:19:58.905778 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:19:58.905782 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:19:58.905843 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:19:58.905992 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:19:58.907485 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:19:58.908273 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:19:58.909575 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:19:58.910042 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:19:58.910156 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:19:58.912721 (+ 2565us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:19:58.912736 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:19:58.912753 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:19:58.912789 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:19:58.915389 (+ 2600us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:19:58.916039 (+ 650us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:19:58.916046 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:19:58.916048 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:19:58.916198 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:19:58.916671 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:19:58.916674 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:19:58.916676 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:19:58.917001 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:19:58.917228 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0620 20:19:58.917551 (+ 323us) server_negotiation.cc:299] Negotiation successful
0620 20:19:58.917711 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260620 20:19:58.918848 1290 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:19:58.979849 1306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:19:58.979303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.1:38751 (local address 127.0.0.1:34142)
0620 20:19:58.979547 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0620 20:19:58.979664 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.217.1:38751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260620 20:19:58.980957 1306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:19:58.980552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.1:38751 (local address 127.0.0.1:34144)
0620 20:19:58.980777 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0620 20:19:58.980847 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.217.1:38751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
I20260620 20:19:59.008129 17252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu with pid 1167
I20260620 20:19:59.026400 17252 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu pid 813
I20260620 20:19:59.027848 876 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:19:59.143962 17252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu with pid 813
I20260620 20:20:19.166903 17252 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
/tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986536687152-17252-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986536687152-17252-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986536687152-17252-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986536687152-17252-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.217.62:34019
--webserver_interface=127.16.217.62
--webserver_port=35859
--keytab_file=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.62.keytab
--principal=kudu/127.16.217.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.217.20:35087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.217.62:34019
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986536687152-17252-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986536687152-17252-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986536687152-17252-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:20:19.499775 1343 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:20:19.500064 1343 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:20:19.500237 1343 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:20:19.507477 1343 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260620 20:20:19.507666 1343 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260620 20:20:19.507743 1343 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260620 20:20:19.507833 1343 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:20:19.507936 1343 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:20:19.508062 1343 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:20:19.508162 1343 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:20:19.508248 1343 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:20:19.518038 1343 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.217.20:35087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986536687152-17252-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986536687152-17252-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.16.217.62:34019
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.62.keytab
--principal=kudu/127.16.217.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.217.62:34019
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986536687152-17252-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.217.62
--webserver_port=35859
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986536687152-17252-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:20:19.519335 1343 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:20:19.520715 1343 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:20:19.533103 1349 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:20:19.533169 1348 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:20:19.533211 1351 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:20:19.533747 1343 server_base.cc:1034] running on GCE node
I20260620 20:20:19.534602 1343 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:20:19.536023 1343 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:20:19.537338 1343 hybrid_clock.cc:648] HybridClock initialized: now 1781986819537252 us; error 58 us; skew 500 ppm
I20260620 20:20:19.537719 1343 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:20:19 dist-test-slave-2hpg krb5kdc[32730](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986819, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.217.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:20:19.540951 1343 init.cc:377] Logged in from keytab as kudu/127.16.217.62@KRBTEST.COM (short username kudu)
I20260620 20:20:19.542766 1343 webserver.cc:460] Webserver started at http://127.16.217.62:35859/ using document root <none> and password file <none>
I20260620 20:20:19.543535 1343 fs_manager.cc:359] Metadata directory not provided
I20260620 20:20:19.543654 1343 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:20:19.549108 1343 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 20:20:19.552024 1358 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:20:19.553537 1343 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 20:20:19.553723 1343 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986536687152-17252-0/minicluster-data/master-0/data,/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986536687152-17252-0/minicluster-data/master-0/wal
uuid: "8e39d1abbd4e46a7bc73a63bfb83dfd0"
format_stamp: "Formatted at 2026-06-20 20:19:33 on dist-test-slave-2hpg"
server_key: "e31576e9a31de7cbfbaa3f0936591892"
server_key_iv: "1cea50ed55f4321d6496cb1ca04baf58"
server_key_version: "encryptionkey@0"
I20260620 20:20:19.554684 1343 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986536687152-17252-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986536687152-17252-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986536687152-17252-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:20:19.570343 1343 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:20:19.574683 1343 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:20:19.575028 1343 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:20:19.595836 1343 rpc_server.cc:307] RPC server started. Bound to: 127.16.217.62:34019
I20260620 20:20:19.595855 1409 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.217.62:34019 every 8 connection(s)
I20260620 20:20:19.597503 1343 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986536687152-17252-0/minicluster-data/master-0/data/info.pb
I20260620 20:20:19.600634 17252 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu as pid 1343
I20260620 20:20:19.601675 17252 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
/tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986536687152-17252-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986536687152-17252-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986536687152-17252-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986536687152-17252-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.217.1:38751
--local_ip_for_outbound_sockets=127.16.217.1
--tserver_master_addrs=127.16.217.62:34019
--webserver_port=46685
--webserver_interface=127.16.217.1
--keytab_file=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.1.keytab
--principal=kudu/127.16.217.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.217.20:35087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986536687152-17252-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986536687152-17252-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986536687152-17252-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 20:20:19.606607 1410 sys_catalog.cc:263] Verifying existing consensus state
I20260620 20:20:19.611545 1410 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8e39d1abbd4e46a7bc73a63bfb83dfd0: Bootstrap starting.
I20260620 20:20:19.632264 1410 log.cc:824] T 00000000000000000000000000000000 P 8e39d1abbd4e46a7bc73a63bfb83dfd0: Log is configured to *not* fsync() on all Append() calls
I20260620 20:20:19.648113 1410 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8e39d1abbd4e46a7bc73a63bfb83dfd0: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 20:20:19.648669 1410 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8e39d1abbd4e46a7bc73a63bfb83dfd0: Bootstrap complete.
I20260620 20:20:19.658321 1410 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8e39d1abbd4e46a7bc73a63bfb83dfd0 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e39d1abbd4e46a7bc73a63bfb83dfd0" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 34019 } }
I20260620 20:20:19.659384 1410 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8e39d1abbd4e46a7bc73a63bfb83dfd0 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e39d1abbd4e46a7bc73a63bfb83dfd0, State: Initialized, Role: FOLLOWER
I20260620 20:20:19.660020 1410 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8e39d1abbd4e46a7bc73a63bfb83dfd0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e39d1abbd4e46a7bc73a63bfb83dfd0" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 34019 } }
I20260620 20:20:19.660274 1410 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8e39d1abbd4e46a7bc73a63bfb83dfd0 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 20:20:19.660416 1410 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8e39d1abbd4e46a7bc73a63bfb83dfd0 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 20:20:19.660614 1410 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8e39d1abbd4e46a7bc73a63bfb83dfd0 [term 2 FOLLOWER]: Advancing to term 3
I20260620 20:20:19.664395 1410 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8e39d1abbd4e46a7bc73a63bfb83dfd0 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e39d1abbd4e46a7bc73a63bfb83dfd0" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 34019 } }
I20260620 20:20:19.664912 1410 leader_election.cc:302] T 00000000000000000000000000000000 P 8e39d1abbd4e46a7bc73a63bfb83dfd0 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8e39d1abbd4e46a7bc73a63bfb83dfd0; no voters:
I20260620 20:20:19.665634 1410 leader_election.cc:288] T 00000000000000000000000000000000 P 8e39d1abbd4e46a7bc73a63bfb83dfd0 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260620 20:20:19.665774 1414 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8e39d1abbd4e46a7bc73a63bfb83dfd0 [term 3 FOLLOWER]: Leader election won for term 3
I20260620 20:20:19.666972 1414 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8e39d1abbd4e46a7bc73a63bfb83dfd0 [term 3 LEADER]: Becoming Leader. State: Replica: 8e39d1abbd4e46a7bc73a63bfb83dfd0, State: Running, Role: LEADER
I20260620 20:20:19.667558 1414 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8e39d1abbd4e46a7bc73a63bfb83dfd0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e39d1abbd4e46a7bc73a63bfb83dfd0" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 34019 } }
I20260620 20:20:19.667865 1410 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8e39d1abbd4e46a7bc73a63bfb83dfd0 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:20:19.670214 1416 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8e39d1abbd4e46a7bc73a63bfb83dfd0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8e39d1abbd4e46a7bc73a63bfb83dfd0. Latest consensus state: current_term: 3 leader_uuid: "8e39d1abbd4e46a7bc73a63bfb83dfd0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e39d1abbd4e46a7bc73a63bfb83dfd0" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 34019 } } }
I20260620 20:20:19.670583 1416 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8e39d1abbd4e46a7bc73a63bfb83dfd0 [sys.catalog]: This master's current role is: LEADER
I20260620 20:20:19.671182 1415 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8e39d1abbd4e46a7bc73a63bfb83dfd0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "8e39d1abbd4e46a7bc73a63bfb83dfd0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e39d1abbd4e46a7bc73a63bfb83dfd0" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 34019 } } }
I20260620 20:20:19.671486 1415 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8e39d1abbd4e46a7bc73a63bfb83dfd0 [sys.catalog]: This master's current role is: LEADER
I20260620 20:20:19.679219 1421 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 20:20:19.690006 1421 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 20:20:19.691975 1421 catalog_manager.cc:1259] Loaded cluster ID: e0dabc224e8647618b3b459dc9e53e35
I20260620 20:20:19.692099 1421 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 20:20:19.697476 1421 catalog_manager.cc:1504] Loading token signing keys...
I20260620 20:20:19.699724 1421 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 8e39d1abbd4e46a7bc73a63bfb83dfd0: Loaded TSK: 0
I20260620 20:20:19.699903 1421 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 8e39d1abbd4e46a7bc73a63bfb83dfd0: Loaded TSK: 1
F20260620 20:20:19.701439 1421 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7ffa62ddc9ad google::LogMessage::Fail() at ??:0
@ 0x7ffa62ddf88a google::LogMessage::SendToLog() at ??:0
@ 0x7ffa62ddc4de google::LogMessage::Flush() at ??:0
@ 0x7ffa62ddda19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7ffa66b407d2 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7ffa66b2f73e kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7ffa7880f1c0 kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7ffa7880e783 kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7ffa7889a2a7 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_10::operator()() at ??:0
@ 0x7ffa7889a0c4 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7ffa788e45e9 std::function<>::operator()() at ??:0
@ 0x7ffa78812cdd kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7ffa78810c36 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7ffa78898610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7ffa78898439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7ffa65a28345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7ffa65a2b91c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7ffa65a2b799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7ffa65a06eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7ffa68ce06db start_thread at ??:0
@ 0x7ffa61c7661f clone at ??:0
W20260620 20:20:20.016815 1412 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:20:20.017186 1412 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:20:20.017359 1412 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:20:20.025077 1412 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:20:20.025216 1412 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:20:20.025339 1412 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.217.1
I20260620 20:20:20.036285 1412 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.217.20:35087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986536687152-17252-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986536687152-17252-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.1.keytab
--principal=kudu/127.16.217.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.217.1:38751
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986536687152-17252-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.217.1
--webserver_port=46685
--tserver_master_addrs=127.16.217.62:34019
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.217.1
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986536687152-17252-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:20:20.037739 1412 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:20:20.039219 1412 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:20:20.052688 1457 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:20:20.052682 1458 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:20:20.052942 1460 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:20:20.053916 1412 server_base.cc:1034] running on GCE node
I20260620 20:20:20.054512 1412 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:20:20.055474 1412 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:20:20.056717 1412 hybrid_clock.cc:648] HybridClock initialized: now 1781986820056665 us; error 45 us; skew 500 ppm
I20260620 20:20:20.057072 1412 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:20:20 dist-test-slave-2hpg krb5kdc[32730](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986820, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.217.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:20:20.060410 1412 init.cc:377] Logged in from keytab as kudu/127.16.217.1@KRBTEST.COM (short username kudu)
I20260620 20:20:20.061848 1412 webserver.cc:460] Webserver started at http://127.16.217.1:46685/ using document root <none> and password file <none>
I20260620 20:20:20.062590 1412 fs_manager.cc:359] Metadata directory not provided
I20260620 20:20:20.062750 1412 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:20:20.067868 1412 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.002s
I20260620 20:20:20.070762 1467 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:20:20.072129 1412 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 20:20:20.072314 1412 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986536687152-17252-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986536687152-17252-0/minicluster-data/ts-0/wal
uuid: "48ddf12e1ab742819abba58564f9d6b5"
format_stamp: "Formatted at 2026-06-20 20:19:34 on dist-test-slave-2hpg"
server_key: "a59ce49777766b69fa009b135592f350"
server_key_iv: "f28fb0e2561f6d706f9cf30682c90572"
server_key_version: "encryptionkey@0"
I20260620 20:20:20.073257 1412 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986536687152-17252-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986536687152-17252-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986536687152-17252-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:20:20.110285 1412 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:20:20.114943 1412 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:20:20.115243 1412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:20:20.116272 1412 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:20:20.118209 1474 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260620 20:20:20.123842 1412 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260620 20:20:20.123991 1412 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.001s sys 0.000s
I20260620 20:20:20.124157 1412 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260620 20:20:20.127269 1412 ts_tablet_manager.cc:610] Registered 1 tablets
I20260620 20:20:20.127374 1412 ts_tablet_manager.cc:589] Time spent register tablets: real 0.003s user 0.000s sys 0.002s
I20260620 20:20:20.127743 1474 tablet_bootstrap.cc:492] T c82c3f55f5654f5aa0d28a92481d18de P 48ddf12e1ab742819abba58564f9d6b5: Bootstrap starting.
I20260620 20:20:20.156172 1474 log.cc:824] T c82c3f55f5654f5aa0d28a92481d18de P 48ddf12e1ab742819abba58564f9d6b5: Log is configured to *not* fsync() on all Append() calls
I20260620 20:20:20.170195 1412 rpc_server.cc:307] RPC server started. Bound to: 127.16.217.1:38751
I20260620 20:20:20.170284 1581 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.217.1:38751 every 8 connection(s)
I20260620 20:20:20.170719 1474 tablet_bootstrap.cc:492] T c82c3f55f5654f5aa0d28a92481d18de P 48ddf12e1ab742819abba58564f9d6b5: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 20:20:20.171191 1474 tablet_bootstrap.cc:492] T c82c3f55f5654f5aa0d28a92481d18de P 48ddf12e1ab742819abba58564f9d6b5: Bootstrap complete.
I20260620 20:20:20.171622 1412 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986536687152-17252-0/minicluster-data/ts-0/data/info.pb
I20260620 20:20:20.172822 1474 ts_tablet_manager.cc:1397] T c82c3f55f5654f5aa0d28a92481d18de P 48ddf12e1ab742819abba58564f9d6b5: Time spent bootstrapping tablet: real 0.045s user 0.026s sys 0.016s
I20260620 20:20:20.177008 1584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:20.174793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:34019 (local address 127.16.217.1:53671)
0620 20:20:20.175738 (+ 945us) negotiation.cc:107] Waiting for socket to connect
0620 20:20:20.176076 (+ 338us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.217.62:34019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":641,"thread_start_us":451,"threads_started":1}
W20260620 20:20:20.177796 1582 heartbeater.cc:644] Failed to heartbeat to 127.16.217.62:34019 (0 consecutive failures): Network error: Failed to ping master at 127.16.217.62:34019: Client connection negotiation failed: client connection to 127.16.217.62:34019: connect: Connection refused (error 111)
I20260620 20:20:20.178756 1584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:20.178385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:34019 (local address 127.16.217.1:52853)
0620 20:20:20.178547 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0620 20:20:20.178646 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.217.62:34019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260620 20:20:20.179724 1584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:20.179296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:34019 (local address 127.16.217.1:56641)
0620 20:20:20.179457 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0620 20:20:20.179558 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.217.62:34019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260620 20:20:20.180119 1582 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 20:20:20.180528 17252 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu as pid 1412
I20260620 20:20:20.181325 17252 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
/tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986536687152-17252-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986536687152-17252-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986536687152-17252-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986536687152-17252-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.217.2:33863
--local_ip_for_outbound_sockets=127.16.217.2
--tserver_master_addrs=127.16.217.62:34019
--webserver_port=44433
--webserver_interface=127.16.217.2
--keytab_file=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.2.keytab
--principal=kudu/127.16.217.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.217.20:35087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986536687152-17252-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986536687152-17252-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986536687152-17252-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 20:20:20.181668 1474 raft_consensus.cc:348] T c82c3f55f5654f5aa0d28a92481d18de P 48ddf12e1ab742819abba58564f9d6b5 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48ddf12e1ab742819abba58564f9d6b5" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 38751 } } peers { permanent_uuid: "24811d2b3eec4754a9194eabaa73b41f" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 46007 } } peers { permanent_uuid: "bb9261ea26ae46a580415794f87a5ee0" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 33863 } }
I20260620 20:20:20.183179 1474 raft_consensus.cc:729] T c82c3f55f5654f5aa0d28a92481d18de P 48ddf12e1ab742819abba58564f9d6b5 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48ddf12e1ab742819abba58564f9d6b5, State: Initialized, Role: FOLLOWER
I20260620 20:20:20.183846 1474 consensus_queue.cc:260] T c82c3f55f5654f5aa0d28a92481d18de P 48ddf12e1ab742819abba58564f9d6b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48ddf12e1ab742819abba58564f9d6b5" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 38751 } } peers { permanent_uuid: "24811d2b3eec4754a9194eabaa73b41f" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 46007 } } peers { permanent_uuid: "bb9261ea26ae46a580415794f87a5ee0" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 33863 } }
I20260620 20:20:20.185535 1584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:20.185189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:34019 (local address 127.16.217.1:48205)
0620 20:20:20.185344 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0620 20:20:20.185423 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.217.62:34019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260620 20:20:20.185616 1474 ts_tablet_manager.cc:1428] T c82c3f55f5654f5aa0d28a92481d18de P 48ddf12e1ab742819abba58564f9d6b5: Time spent starting tablet: real 0.012s user 0.011s sys 0.000s
I20260620 20:20:20.186566 1584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:20.186227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:34019 (local address 127.16.217.1:56629)
0620 20:20:20.186376 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0620 20:20:20.186453 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.217.62:34019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260620 20:20:20.508466 1586 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:20:20.508777 1586 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:20:20.508942 1586 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:20:20.516147 1586 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:20:20.516300 1586 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:20:20.516470 1586 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.217.2
I20260620 20:20:20.526096 1586 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.217.20:35087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986536687152-17252-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986536687152-17252-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.2.keytab
--principal=kudu/127.16.217.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.217.2:33863
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986536687152-17252-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.217.2
--webserver_port=44433
--tserver_master_addrs=127.16.217.62:34019
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.217.2
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986536687152-17252-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:20:20.527348 1586 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:20:20.528632 1586 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:20:20.541646 1594 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:20:20.541690 1592 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:20:20.541721 1591 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:20:20.542618 1586 server_base.cc:1034] running on GCE node
I20260620 20:20:20.543201 1586 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:20:20.544092 1586 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:20:20.545315 1586 hybrid_clock.cc:648] HybridClock initialized: now 1781986820545278 us; error 51 us; skew 500 ppm
I20260620 20:20:20.545617 1586 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:20:20 dist-test-slave-2hpg krb5kdc[32730](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986820, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.217.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:20:20.548731 1586 init.cc:377] Logged in from keytab as kudu/127.16.217.2@KRBTEST.COM (short username kudu)
I20260620 20:20:20.550262 1586 webserver.cc:460] Webserver started at http://127.16.217.2:44433/ using document root <none> and password file <none>
I20260620 20:20:20.550927 1586 fs_manager.cc:359] Metadata directory not provided
I20260620 20:20:20.551079 1586 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:20:20.556300 1586 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 20:20:20.558955 1601 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:20:20.560314 1586 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 20:20:20.560504 1586 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986536687152-17252-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986536687152-17252-0/minicluster-data/ts-1/wal
uuid: "bb9261ea26ae46a580415794f87a5ee0"
format_stamp: "Formatted at 2026-06-20 20:19:34 on dist-test-slave-2hpg"
server_key: "7b9d4c4e8c73a23e07f972a525e02b22"
server_key_iv: "caa256f0c4f6f0763ef305f7f306aecd"
server_key_version: "encryptionkey@0"
I20260620 20:20:20.561507 1586 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986536687152-17252-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986536687152-17252-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986536687152-17252-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:20:20.595577 1586 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:20:20.600157 1586 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:20:20.600430 1586 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:20:20.601461 1586 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:20:20.602835 1586 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:20:20.602954 1586 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:20:20.603096 1586 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:20:20.603183 1586 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:20:20.636806 1586 rpc_server.cc:307] RPC server started. Bound to: 127.16.217.2:33863
I20260620 20:20:20.636897 1713 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.217.2:33863 every 8 connection(s)
I20260620 20:20:20.638346 1586 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986536687152-17252-0/minicluster-data/ts-1/data/info.pb
I20260620 20:20:20.643081 1716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:20.641224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:34019 (local address 127.16.217.2:37171)
0620 20:20:20.642006 (+ 782us) negotiation.cc:107] Waiting for socket to connect
0620 20:20:20.642252 (+ 246us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.217.62:34019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":470,"thread_start_us":281,"threads_started":1}
W20260620 20:20:20.643738 1714 heartbeater.cc:644] Failed to heartbeat to 127.16.217.62:34019 (0 consecutive failures): Network error: Failed to ping master at 127.16.217.62:34019: Client connection negotiation failed: client connection to 127.16.217.62:34019: connect: Connection refused (error 111)
I20260620 20:20:20.644492 1716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:20.644197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:34019 (local address 127.16.217.2:45195)
0620 20:20:20.644327 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0620 20:20:20.644391 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.217.62:34019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260620 20:20:20.645265 1716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:20.644967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:34019 (local address 127.16.217.2:38215)
0620 20:20:20.645084 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0620 20:20:20.645162 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.217.62:34019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260620 20:20:20.645488 1714 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 20:20:20.646574 17252 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu as pid 1586
I20260620 20:20:20.647226 17252 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
/tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986536687152-17252-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986536687152-17252-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986536687152-17252-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986536687152-17252-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.217.3:46007
--local_ip_for_outbound_sockets=127.16.217.3
--tserver_master_addrs=127.16.217.62:34019
--webserver_port=38299
--webserver_interface=127.16.217.3
--keytab_file=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.3.keytab
--principal=kudu/127.16.217.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.217.20:35087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986536687152-17252-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986536687152-17252-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986536687152-17252-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:20:20.976105 1717 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:20:20.976404 1717 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:20:20.976503 1717 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:20:20.983973 1717 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:20:20.984078 1717 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:20:20.984190 1717 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.217.3
I20260620 20:20:20.993891 1717 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.217.20:35087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986536687152-17252-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986536687152-17252-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.3.keytab
--principal=kudu/127.16.217.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.217.3:46007
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986536687152-17252-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.217.3
--webserver_port=38299
--tserver_master_addrs=127.16.217.62:34019
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.217.3
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986536687152-17252-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:20:20.995097 1717 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:20:20.996304 1717 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:20:21.009400 1722 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:20:21.009466 1723 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:20:21.009441 1725 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:20:21.010458 1717 server_base.cc:1034] running on GCE node
I20260620 20:20:21.011260 1717 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:20:21.012344 1717 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:20:21.013571 1717 hybrid_clock.cc:648] HybridClock initialized: now 1781986821013521 us; error 36 us; skew 500 ppm
I20260620 20:20:21.013990 1717 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:20:21 dist-test-slave-2hpg krb5kdc[32730](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986821, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.217.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:20:21.025478 1717 init.cc:377] Logged in from keytab as kudu/127.16.217.3@KRBTEST.COM (short username kudu)
I20260620 20:20:21.027148 1717 webserver.cc:460] Webserver started at http://127.16.217.3:38299/ using document root <none> and password file <none>
I20260620 20:20:21.027864 1717 fs_manager.cc:359] Metadata directory not provided
I20260620 20:20:21.028012 1717 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:20:21.033331 1717 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260620 20:20:21.037447 1732 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:20:21.039005 1717 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 20:20:21.039218 1717 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986536687152-17252-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986536687152-17252-0/minicluster-data/ts-2/wal
uuid: "24811d2b3eec4754a9194eabaa73b41f"
format_stamp: "Formatted at 2026-06-20 20:19:35 on dist-test-slave-2hpg"
server_key: "4d2eb5a7b0d6dc44451589d8f4e00b44"
server_key_iv: "cf7eb6dbad246eff8cf648f969b186d1"
server_key_version: "encryptionkey@0"
I20260620 20:20:21.040266 1717 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986536687152-17252-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986536687152-17252-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986536687152-17252-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:20:21.076145 1717 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:20:21.081058 1717 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:20:21.081379 1717 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:20:21.082520 1717 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:20:21.083962 1717 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:20:21.084101 1717 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:20:21.084247 1717 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:20:21.084362 1717 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:20:21.120209 1717 rpc_server.cc:307] RPC server started. Bound to: 127.16.217.3:46007
I20260620 20:20:21.120307 1844 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.217.3:46007 every 8 connection(s)
I20260620 20:20:21.121881 1717 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986536687152-17252-0/minicluster-data/ts-2/data/info.pb
I20260620 20:20:21.126935 1847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:21.124919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:34019 (local address 127.16.217.3:58937)
0620 20:20:21.125705 (+ 786us) negotiation.cc:107] Waiting for socket to connect
0620 20:20:21.126023 (+ 318us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.217.62:34019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":478,"thread_start_us":292,"threads_started":1}
W20260620 20:20:21.127820 1845 heartbeater.cc:644] Failed to heartbeat to 127.16.217.62:34019 (0 consecutive failures): Network error: Failed to ping master at 127.16.217.62:34019: Client connection negotiation failed: client connection to 127.16.217.62:34019: connect: Connection refused (error 111)
I20260620 20:20:21.128870 1847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:21.128419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:34019 (local address 127.16.217.3:45575)
0620 20:20:21.128687 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0620 20:20:21.128761 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.217.62:34019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160}
I20260620 20:20:21.129801 1847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:21.129413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:34019 (local address 127.16.217.3:45775)
0620 20:20:21.129599 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0620 20:20:21.129684 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.217.62:34019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
W20260620 20:20:21.130193 1845 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 20:20:21.131716 17252 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu as pid 1717
I20260620 20:20:21.188314 1849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:21.187518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:34019 (local address 127.16.217.1:49415)
0620 20:20:21.188016 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0620 20:20:21.188110 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.217.62:34019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":396,"thread_start_us":222,"threads_started":1}
I20260620 20:20:21.562186 1850 raft_consensus.cc:482] T c82c3f55f5654f5aa0d28a92481d18de P 48ddf12e1ab742819abba58564f9d6b5 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 20:20:21.562518 1850 raft_consensus.cc:504] T c82c3f55f5654f5aa0d28a92481d18de P 48ddf12e1ab742819abba58564f9d6b5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48ddf12e1ab742819abba58564f9d6b5" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 38751 } } peers { permanent_uuid: "24811d2b3eec4754a9194eabaa73b41f" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 46007 } } peers { permanent_uuid: "bb9261ea26ae46a580415794f87a5ee0" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 33863 } }
I20260620 20:20:21.564241 1850 leader_election.cc:288] T c82c3f55f5654f5aa0d28a92481d18de P 48ddf12e1ab742819abba58564f9d6b5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 24811d2b3eec4754a9194eabaa73b41f (127.16.217.3:46007), bb9261ea26ae46a580415794f87a5ee0 (127.16.217.2:33863)
Jun 20 20:20:21 dist-test-slave-2hpg krb5kdc[32730](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986820, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.217.1@KRBTEST.COM for kudu/127.16.217.3@KRBTEST.COM
Jun 20 20:20:21 dist-test-slave-2hpg krb5kdc[32730](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986820, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.217.1@KRBTEST.COM for kudu/127.16.217.2@KRBTEST.COM
I20260620 20:20:21.587584 1853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:21.564887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.217.1:54291 (local address 127.16.217.3:46007)
0620 20:20:21.566739 (+ 1852us) server_negotiation.cc:206] Beginning negotiation
0620 20:20:21.566757 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 20:20:21.566802 (+ 45us) server_negotiation.cc:407] Connection header received
0620 20:20:21.569090 (+ 2288us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:20:21.569102 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:20:21.569615 (+ 513us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:20:21.569998 (+ 383us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:20:21.572034 (+ 2036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:21.573205 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:21.574903 (+ 1698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:21.575551 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:21.575754 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:20:21.579983 (+ 4229us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:20:21.579990 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:20:21.580005 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:20:21.580084 (+ 79us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:20:21.583524 (+ 3440us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:20:21.584463 (+ 939us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:20:21.584473 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:20:21.584482 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:20:21.584654 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:20:21.585114 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:20:21.585122 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:20:21.585125 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:20:21.585691 (+ 566us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:20:21.585921 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0620 20:20:21.586778 (+ 857us) server_negotiation.cc:299] Negotiation successful
0620 20:20:21.587151 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1644,"thread_start_us":1514,"threads_started":1}
I20260620 20:20:21.587617 1851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:21.564637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.2:33863 (local address 127.16.217.1:47209)
0620 20:20:21.566404 (+ 1767us) negotiation.cc:107] Waiting for socket to connect
0620 20:20:21.566423 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 20:20:21.566533 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:20:21.570208 (+ 3675us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:20:21.570220 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:20:21.571082 (+ 862us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:20:21.571845 (+ 763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:20:21.571866 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:20:21.573453 (+ 1587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:20:21.573459 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:20:21.574759 (+ 1300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:20:21.574770 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:20:21.575820 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:20:21.575828 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:20:21.576046 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:20:21.576986 (+ 940us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:20:21.577003 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:20:21.579986 (+ 2983us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:20:21.583854 (+ 3868us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:20:21.583858 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:20:21.583860 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:20:21.584333 (+ 473us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:20:21.584843 (+ 510us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:20:21.584847 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:20:21.584849 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:20:21.585006 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:20:21.585820 (+ 814us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:20:21.585828 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:20:21.586256 (+ 428us) client_negotiation.cc:769] Sending connection context
0620 20:20:21.586559 (+ 303us) client_negotiation.cc:240] Negotiation successful
0620 20:20:21.587143 (+ 584us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1640,"mutex_wait_us":60,"spinlock_wait_cycles":8320,"thread_start_us":1471,"threads_started":1}
I20260620 20:20:21.587651 1852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:21.564854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.217.1:47209 (local address 127.16.217.2:33863)
0620 20:20:21.566975 (+ 2121us) server_negotiation.cc:206] Beginning negotiation
0620 20:20:21.566989 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 20:20:21.567020 (+ 31us) server_negotiation.cc:407] Connection header received
0620 20:20:21.569092 (+ 2072us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:20:21.569111 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:20:21.569610 (+ 499us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:20:21.569998 (+ 388us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:20:21.572036 (+ 2038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:21.573297 (+ 1261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:21.574895 (+ 1598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:21.575564 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:21.575775 (+ 211us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:20:21.580269 (+ 4494us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:20:21.580281 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:20:21.580298 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:20:21.580342 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:20:21.583692 (+ 3350us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:20:21.584512 (+ 820us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:20:21.584518 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:20:21.584524 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:20:21.584698 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:20:21.585161 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:20:21.585165 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:20:21.585167 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:20:21.585675 (+ 508us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:20:21.585870 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0620 20:20:21.586803 (+ 933us) server_negotiation.cc:299] Negotiation successful
0620 20:20:21.587214 (+ 411us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1953,"thread_start_us":1847,"threads_started":1}
I20260620 20:20:21.587723 1849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:21.564528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.3:46007 (local address 127.16.217.1:54291)
0620 20:20:21.564791 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0620 20:20:21.564827 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 20:20:21.565547 (+ 720us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:20:21.570606 (+ 5059us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:20:21.570613 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:20:21.571116 (+ 503us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:20:21.571845 (+ 729us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:20:21.571865 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:20:21.573388 (+ 1523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:20:21.573392 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:20:21.574765 (+ 1373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:20:21.574777 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:20:21.575772 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:20:21.575778 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:20:21.576044 (+ 266us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:20:21.576897 (+ 853us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:20:21.576920 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:20:21.579670 (+ 2750us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:20:21.583692 (+ 4022us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:20:21.583702 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:20:21.583717 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:20:21.584309 (+ 592us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:20:21.584780 (+ 471us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:20:21.584787 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:20:21.584793 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:20:21.584997 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:20:21.585855 (+ 858us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:20:21.585860 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:20:21.586280 (+ 420us) client_negotiation.cc:769] Sending connection context
0620 20:20:21.586560 (+ 280us) client_negotiation.cc:240] Negotiation successful
0620 20:20:21.587300 (+ 740us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"mutex_wait_us":310,"spinlock_wait_cycles":12928}
W20260620 20:20:21.591097 1469 leader_election.cc:341] T c82c3f55f5654f5aa0d28a92481d18de P 48ddf12e1ab742819abba58564f9d6b5 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 24811d2b3eec4754a9194eabaa73b41f (127.16.217.3:46007): Not found: Tablet not found: c82c3f55f5654f5aa0d28a92481d18de
W20260620 20:20:21.591398 1470 leader_election.cc:341] T c82c3f55f5654f5aa0d28a92481d18de P 48ddf12e1ab742819abba58564f9d6b5 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer bb9261ea26ae46a580415794f87a5ee0 (127.16.217.2:33863): Not found: Tablet not found: c82c3f55f5654f5aa0d28a92481d18de
I20260620 20:20:21.591584 1470 leader_election.cc:302] T c82c3f55f5654f5aa0d28a92481d18de P 48ddf12e1ab742819abba58564f9d6b5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 48ddf12e1ab742819abba58564f9d6b5; no voters: 24811d2b3eec4754a9194eabaa73b41f, bb9261ea26ae46a580415794f87a5ee0
I20260620 20:20:21.592025 1850 raft_consensus.cc:2726] T c82c3f55f5654f5aa0d28a92481d18de P 48ddf12e1ab742819abba58564f9d6b5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 20:20:21.646986 1854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:21.646056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:34019 (local address 127.16.217.2:42871)
0620 20:20:21.646539 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0620 20:20:21.646737 (+ 198us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.217.62:34019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":228,"threads_started":1}
I20260620 20:20:22.131572 1855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:22.130755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:34019 (local address 127.16.217.3:39085)
0620 20:20:22.131227 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0620 20:20:22.131348 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.217.62:34019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":351,"thread_start_us":225,"threads_started":1}
I20260620 20:20:22.190220 1856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:22.189464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:34019 (local address 127.16.217.1:41045)
0620 20:20:22.189965 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0620 20:20:22.190050 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.217.62:34019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":338,"thread_start_us":238,"threads_started":1}
I20260620 20:20:22.648684 1857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:22.647953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:34019 (local address 127.16.217.2:56991)
0620 20:20:22.648427 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0620 20:20:22.648517 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.217.62:34019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":372,"thread_start_us":268,"threads_started":1}
I20260620 20:20:23.133309 1858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:23.132514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:34019 (local address 127.16.217.3:50025)
0620 20:20:23.132948 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0620 20:20:23.133089 (+ 141us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.217.62:34019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":341,"thread_start_us":234,"threads_started":1}
I20260620 20:20:23.192411 1859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:23.191295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:34019 (local address 127.16.217.1:33619)
0620 20:20:23.191993 (+ 698us) negotiation.cc:107] Waiting for socket to connect
0620 20:20:23.192124 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.217.62:34019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":553,"thread_start_us":323,"threads_started":1}
I20260620 20:20:23.650352 1860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:23.649597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:34019 (local address 127.16.217.2:52605)
0620 20:20:23.650085 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0620 20:20:23.650187 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.217.62:34019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":372,"thread_start_us":224,"threads_started":1}
I20260620 20:20:23.830489 1861 raft_consensus.cc:482] T c82c3f55f5654f5aa0d28a92481d18de P 48ddf12e1ab742819abba58564f9d6b5 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 20:20:23.830700 1861 raft_consensus.cc:504] T c82c3f55f5654f5aa0d28a92481d18de P 48ddf12e1ab742819abba58564f9d6b5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48ddf12e1ab742819abba58564f9d6b5" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 38751 } } peers { permanent_uuid: "24811d2b3eec4754a9194eabaa73b41f" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 46007 } } peers { permanent_uuid: "bb9261ea26ae46a580415794f87a5ee0" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 33863 } }
I20260620 20:20:23.831384 1861 leader_election.cc:288] T c82c3f55f5654f5aa0d28a92481d18de P 48ddf12e1ab742819abba58564f9d6b5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 24811d2b3eec4754a9194eabaa73b41f (127.16.217.3:46007), bb9261ea26ae46a580415794f87a5ee0 (127.16.217.2:33863)
W20260620 20:20:23.832362 1469 leader_election.cc:341] T c82c3f55f5654f5aa0d28a92481d18de P 48ddf12e1ab742819abba58564f9d6b5 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 24811d2b3eec4754a9194eabaa73b41f (127.16.217.3:46007): Not found: Tablet not found: c82c3f55f5654f5aa0d28a92481d18de
W20260620 20:20:23.832573 1470 leader_election.cc:341] T c82c3f55f5654f5aa0d28a92481d18de P 48ddf12e1ab742819abba58564f9d6b5 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer bb9261ea26ae46a580415794f87a5ee0 (127.16.217.2:33863): Not found: Tablet not found: c82c3f55f5654f5aa0d28a92481d18de
I20260620 20:20:23.832698 1470 leader_election.cc:302] T c82c3f55f5654f5aa0d28a92481d18de P 48ddf12e1ab742819abba58564f9d6b5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 48ddf12e1ab742819abba58564f9d6b5; no voters: 24811d2b3eec4754a9194eabaa73b41f, bb9261ea26ae46a580415794f87a5ee0
I20260620 20:20:23.832950 1861 raft_consensus.cc:2726] T c82c3f55f5654f5aa0d28a92481d18de P 48ddf12e1ab742819abba58564f9d6b5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 20:20:24.134922 1862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:24.134205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:34019 (local address 127.16.217.3:43133)
0620 20:20:24.134655 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0620 20:20:24.134754 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.217.62:34019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":237,"threads_started":1}
I20260620 20:20:24.194252 1863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:24.193421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:34019 (local address 127.16.217.1:57085)
0620 20:20:24.193961 (+ 540us) negotiation.cc:107] Waiting for socket to connect
0620 20:20:24.194048 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.217.62:34019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":246,"threads_started":1}
I20260620 20:20:24.652307 1864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:24.651388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:34019 (local address 127.16.217.2:51113)
0620 20:20:24.652000 (+ 612us) negotiation.cc:107] Waiting for socket to connect
0620 20:20:24.652125 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.217.62:34019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":500,"thread_start_us":286,"threads_started":1}
I20260620 20:20:25.136646 1865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:25.135863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:34019 (local address 127.16.217.3:51269)
0620 20:20:25.136338 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0620 20:20:25.136421 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.217.62:34019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":248,"threads_started":1}
I20260620 20:20:25.196354 1866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:25.195445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:34019 (local address 127.16.217.1:55467)
0620 20:20:25.195957 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0620 20:20:25.196055 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.217.62:34019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":392,"thread_start_us":289,"threads_started":1}
I20260620 20:20:25.654037 1867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:25.653260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:34019 (local address 127.16.217.2:40231)
0620 20:20:25.653727 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0620 20:20:25.653807 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.217.62:34019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":372,"thread_start_us":220,"threads_started":1}
I20260620 20:20:26.138386 1868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:26.137608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:34019 (local address 127.16.217.3:35905)
0620 20:20:26.138134 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0620 20:20:26.138227 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.217.62:34019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":422,"thread_start_us":298,"threads_started":1}
I20260620 20:20:26.198407 1869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:26.197381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:34019 (local address 127.16.217.1:45433)
0620 20:20:26.198095 (+ 714us) negotiation.cc:107] Waiting for socket to connect
0620 20:20:26.198214 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.217.62:34019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":550,"thread_start_us":359,"threads_started":1}
I20260620 20:20:26.325714 1870 raft_consensus.cc:482] T c82c3f55f5654f5aa0d28a92481d18de P 48ddf12e1ab742819abba58564f9d6b5 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 20:20:26.326001 1870 raft_consensus.cc:504] T c82c3f55f5654f5aa0d28a92481d18de P 48ddf12e1ab742819abba58564f9d6b5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48ddf12e1ab742819abba58564f9d6b5" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 38751 } } peers { permanent_uuid: "24811d2b3eec4754a9194eabaa73b41f" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 46007 } } peers { permanent_uuid: "bb9261ea26ae46a580415794f87a5ee0" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 33863 } }
I20260620 20:20:26.326702 1870 leader_election.cc:288] T c82c3f55f5654f5aa0d28a92481d18de P 48ddf12e1ab742819abba58564f9d6b5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 24811d2b3eec4754a9194eabaa73b41f (127.16.217.3:46007), bb9261ea26ae46a580415794f87a5ee0 (127.16.217.2:33863)
W20260620 20:20:26.327728 1470 leader_election.cc:341] T c82c3f55f5654f5aa0d28a92481d18de P 48ddf12e1ab742819abba58564f9d6b5 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer bb9261ea26ae46a580415794f87a5ee0 (127.16.217.2:33863): Not found: Tablet not found: c82c3f55f5654f5aa0d28a92481d18de
W20260620 20:20:26.327941 1469 leader_election.cc:341] T c82c3f55f5654f5aa0d28a92481d18de P 48ddf12e1ab742819abba58564f9d6b5 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 24811d2b3eec4754a9194eabaa73b41f (127.16.217.3:46007): Not found: Tablet not found: c82c3f55f5654f5aa0d28a92481d18de
I20260620 20:20:26.328087 1469 leader_election.cc:302] T c82c3f55f5654f5aa0d28a92481d18de P 48ddf12e1ab742819abba58564f9d6b5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 48ddf12e1ab742819abba58564f9d6b5; no voters: 24811d2b3eec4754a9194eabaa73b41f, bb9261ea26ae46a580415794f87a5ee0
I20260620 20:20:26.328377 1870 raft_consensus.cc:2726] T c82c3f55f5654f5aa0d28a92481d18de P 48ddf12e1ab742819abba58564f9d6b5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 20:20:26.655732 1871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:26.655020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:34019 (local address 127.16.217.2:50259)
0620 20:20:26.655476 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0620 20:20:26.655556 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.217.62:34019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":243,"threads_started":1}
I20260620 20:20:27.140127 1872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:27.139360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:34019 (local address 127.16.217.3:48019)
0620 20:20:27.139848 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0620 20:20:27.139927 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.217.62:34019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":398,"thread_start_us":291,"threads_started":1}
I20260620 20:20:27.200600 1873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:27.199528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:34019 (local address 127.16.217.1:48245)
0620 20:20:27.200131 (+ 603us) negotiation.cc:107] Waiting for socket to connect
0620 20:20:27.200309 (+ 178us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.217.62:34019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":477,"thread_start_us":332,"threads_started":1}
I20260620 20:20:27.657449 1874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:27.656614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:34019 (local address 127.16.217.2:46661)
0620 20:20:27.657165 (+ 551us) negotiation.cc:107] Waiting for socket to connect
0620 20:20:27.657255 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.217.62:34019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":254,"threads_started":1}
I20260620 20:20:28.141916 1875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:28.141014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:34019 (local address 127.16.217.3:58899)
0620 20:20:28.141582 (+ 568us) negotiation.cc:107] Waiting for socket to connect
0620 20:20:28.141670 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.217.62:34019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":437,"thread_start_us":268,"threads_started":1}
I20260620 20:20:28.202216 1876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:28.201427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:34019 (local address 127.16.217.1:41881)
0620 20:20:28.201950 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0620 20:20:28.202059 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.217.62:34019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":361,"thread_start_us":249,"threads_started":1}
I20260620 20:20:28.659163 1877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:28.658449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:34019 (local address 127.16.217.2:43849)
0620 20:20:28.658862 (+ 413us) negotiation.cc:107] Waiting for socket to connect
0620 20:20:28.658953 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.217.62:34019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":315,"thread_start_us":214,"threads_started":1}
I20260620 20:20:29.143532 1878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:29.142827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:34019 (local address 127.16.217.3:44189)
0620 20:20:29.143257 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0620 20:20:29.143340 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.217.62:34019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":211,"threads_started":1}
I20260620 20:20:29.203861 1879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:29.203107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:34019 (local address 127.16.217.1:41621)
0620 20:20:29.203547 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0620 20:20:29.203677 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.217.62:34019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":340,"thread_start_us":227,"threads_started":1}
I20260620 20:20:29.660806 1880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:29.660053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:34019 (local address 127.16.217.2:39397)
0620 20:20:29.660564 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0620 20:20:29.660651 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.217.62:34019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":232,"threads_started":1}
I20260620 20:20:30.145305 1881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:30.144360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:34019 (local address 127.16.217.3:49805)
0620 20:20:30.145033 (+ 673us) negotiation.cc:107] Waiting for socket to connect
0620 20:20:30.145126 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.217.62:34019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":495,"thread_start_us":340,"threads_started":1}
I20260620 20:20:30.205631 1882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:30.204936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:34019 (local address 127.16.217.1:41933)
0620 20:20:30.205379 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0620 20:20:30.205459 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.217.62:34019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":235,"threads_started":1}
I20260620 20:20:30.662588 1883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:30.661850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:34019 (local address 127.16.217.2:43911)
0620 20:20:30.662318 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0620 20:20:30.662403 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.217.62:34019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":227,"threads_started":1}
I20260620 20:20:31.147141 1884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:31.146393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:34019 (local address 127.16.217.3:32911)
0620 20:20:31.146875 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0620 20:20:31.146962 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.217.62:34019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":373,"thread_start_us":266,"threads_started":1}
I20260620 20:20:31.207487 1885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:31.206657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:34019 (local address 127.16.217.1:51005)
0620 20:20:31.207217 (+ 560us) negotiation.cc:107] Waiting for socket to connect
0620 20:20:31.207297 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.217.62:34019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":317,"threads_started":1}
I20260620 20:20:31.212777 1886 raft_consensus.cc:482] T c82c3f55f5654f5aa0d28a92481d18de P 48ddf12e1ab742819abba58564f9d6b5 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 20:20:31.213107 1886 raft_consensus.cc:504] T c82c3f55f5654f5aa0d28a92481d18de P 48ddf12e1ab742819abba58564f9d6b5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48ddf12e1ab742819abba58564f9d6b5" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 38751 } } peers { permanent_uuid: "24811d2b3eec4754a9194eabaa73b41f" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 46007 } } peers { permanent_uuid: "bb9261ea26ae46a580415794f87a5ee0" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 33863 } }
I20260620 20:20:31.213963 1886 leader_election.cc:288] T c82c3f55f5654f5aa0d28a92481d18de P 48ddf12e1ab742819abba58564f9d6b5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 24811d2b3eec4754a9194eabaa73b41f (127.16.217.3:46007), bb9261ea26ae46a580415794f87a5ee0 (127.16.217.2:33863)
W20260620 20:20:31.214994 1469 leader_election.cc:341] T c82c3f55f5654f5aa0d28a92481d18de P 48ddf12e1ab742819abba58564f9d6b5 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 24811d2b3eec4754a9194eabaa73b41f (127.16.217.3:46007): Not found: Tablet not found: c82c3f55f5654f5aa0d28a92481d18de
W20260620 20:20:31.215170 1470 leader_election.cc:341] T c82c3f55f5654f5aa0d28a92481d18de P 48ddf12e1ab742819abba58564f9d6b5 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer bb9261ea26ae46a580415794f87a5ee0 (127.16.217.2:33863): Not found: Tablet not found: c82c3f55f5654f5aa0d28a92481d18de
I20260620 20:20:31.215337 1470 leader_election.cc:302] T c82c3f55f5654f5aa0d28a92481d18de P 48ddf12e1ab742819abba58564f9d6b5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 48ddf12e1ab742819abba58564f9d6b5; no voters: 24811d2b3eec4754a9194eabaa73b41f, bb9261ea26ae46a580415794f87a5ee0
I20260620 20:20:31.215657 1886 raft_consensus.cc:2726] T c82c3f55f5654f5aa0d28a92481d18de P 48ddf12e1ab742819abba58564f9d6b5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 20:20:31.664191 1887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:31.663452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:34019 (local address 127.16.217.2:55919)
0620 20:20:31.663903 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0620 20:20:31.663996 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.217.62:34019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":363,"thread_start_us":242,"threads_started":1}
I20260620 20:20:32.148816 1888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:32.148062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:34019 (local address 127.16.217.3:40013)
0620 20:20:32.148575 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0620 20:20:32.148666 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.217.62:34019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":383,"thread_start_us":247,"threads_started":1}
I20260620 20:20:32.209242 1889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:32.208441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:34019 (local address 127.16.217.1:37647)
0620 20:20:32.208987 (+ 546us) negotiation.cc:107] Waiting for socket to connect
0620 20:20:32.209065 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.217.62:34019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":434,"thread_start_us":340,"threads_started":1}
I20260620 20:20:32.665827 1890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:32.665157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:34019 (local address 127.16.217.2:47615)
0620 20:20:32.665579 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0620 20:20:32.665658 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.217.62:34019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":328,"thread_start_us":216,"threads_started":1}
I20260620 20:20:33.150583 1891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:33.149770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:34019 (local address 127.16.217.3:41539)
0620 20:20:33.150293 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0620 20:20:33.150373 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.217.62:34019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":428,"thread_start_us":296,"threads_started":1}
I20260620 20:20:33.211084 1892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:33.210307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:34019 (local address 127.16.217.1:33439)
0620 20:20:33.210784 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0620 20:20:33.210875 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.217.62:34019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":359,"thread_start_us":250,"threads_started":1}
I20260620 20:20:33.667654 1893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:33.666941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:34019 (local address 127.16.217.2:47573)
0620 20:20:33.667371 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0620 20:20:33.667459 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.217.62:34019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":225,"threads_started":1}
I20260620 20:20:34.152415 1894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:34.151684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:34019 (local address 127.16.217.3:59075)
0620 20:20:34.152135 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0620 20:20:34.152225 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.217.62:34019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":233,"threads_started":1}
I20260620 20:20:34.212734 1895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:34.211961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:34019 (local address 127.16.217.1:34539)
0620 20:20:34.212434 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0620 20:20:34.212513 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.217.62:34019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":268,"threads_started":1}
I20260620 20:20:34.669134 1896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:34.668446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:34019 (local address 127.16.217.2:36023)
0620 20:20:34.668856 (+ 410us) negotiation.cc:107] Waiting for socket to connect
0620 20:20:34.668940 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.217.62:34019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":323,"thread_start_us":225,"threads_started":1}
I20260620 20:20:35.154076 1897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:35.153334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:34019 (local address 127.16.217.3:46175)
0620 20:20:35.153756 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0620 20:20:35.153832 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.217.62:34019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":227,"threads_started":1}
I20260620 20:20:35.214754 1898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:35.214003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:34019 (local address 127.16.217.1:54665)
0620 20:20:35.214459 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0620 20:20:35.214566 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.217.62:34019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":225,"threads_started":1}
I20260620 20:20:35.671013 1899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:35.670290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:34019 (local address 127.16.217.2:42575)
0620 20:20:35.670758 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0620 20:20:35.670855 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.217.62:34019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":248,"threads_started":1}
I20260620 20:20:36.135190 17252 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu pid 1412
I20260620 20:20:36.150763 1900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:36.136287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60750 (local address 127.16.217.1:38751)
0620 20:20:36.136885 (+ 598us) server_negotiation.cc:206] Beginning negotiation
0620 20:20:36.136893 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 20:20:36.136918 (+ 25us) server_negotiation.cc:407] Connection header received
0620 20:20:36.136982 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:20:36.136989 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:20:36.137205 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:20:36.137413 (+ 208us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:20:36.138907 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:36.139904 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:36.141147 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:36.141739 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:36.141929 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:20:36.144709 (+ 2780us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:20:36.144721 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:20:36.144736 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:20:36.144776 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:20:36.147788 (+ 3012us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:20:36.148527 (+ 739us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:20:36.148534 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:20:36.148543 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:20:36.148688 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:20:36.149060 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:20:36.149067 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:20:36.149072 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:20:36.149459 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:20:36.149804 (+ 345us) server_negotiation.cc:1035] Waiting for connection context
0620 20:20:36.150122 (+ 318us) server_negotiation.cc:299] Negotiation successful
0620 20:20:36.150409 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":428,"thread_start_us":307,"threads_started":1}
I20260620 20:20:36.152227 1577 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:20:36.156124 1902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:36.155274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:34019 (local address 127.16.217.3:51989)
0620 20:20:36.155775 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0620 20:20:36.155887 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.217.62:34019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":369,"thread_start_us":233,"threads_started":1}
I20260620 20:20:36.244853 17252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu with pid 1412
I20260620 20:20:36.262442 17252 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu pid 1586
I20260620 20:20:36.277338 1904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:36.263295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40776 (local address 127.16.217.2:33863)
0620 20:20:36.263872 (+ 577us) server_negotiation.cc:206] Beginning negotiation
0620 20:20:36.263876 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:20:36.263892 (+ 16us) server_negotiation.cc:407] Connection header received
0620 20:20:36.263951 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:20:36.263955 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:20:36.264015 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:20:36.264174 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:20:36.265672 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:36.266681 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:36.268181 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:36.268664 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:36.268809 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:20:36.271733 (+ 2924us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:20:36.271739 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:20:36.271743 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:20:36.271779 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:20:36.274629 (+ 2850us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:20:36.275326 (+ 697us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:20:36.275339 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:20:36.275342 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:20:36.275475 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:20:36.275858 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:20:36.275864 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:20:36.275867 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:20:36.276151 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:20:36.276435 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0620 20:20:36.276736 (+ 301us) server_negotiation.cc:299] Negotiation successful
0620 20:20:36.276962 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":474,"thread_start_us":333,"threads_started":1}
I20260620 20:20:36.278375 1709 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:20:36.361959 17252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu with pid 1586
I20260620 20:20:36.379724 17252 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu pid 1717
I20260620 20:20:36.393481 1906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:36.380484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39486 (local address 127.16.217.3:46007)
0620 20:20:36.381016 (+ 532us) server_negotiation.cc:206] Beginning negotiation
0620 20:20:36.381021 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:20:36.381036 (+ 15us) server_negotiation.cc:407] Connection header received
0620 20:20:36.381100 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:20:36.381104 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:20:36.381160 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:20:36.381292 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:20:36.382790 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:36.383599 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:36.384966 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:36.385402 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:36.385551 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:20:36.388118 (+ 2567us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:20:36.388123 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:20:36.388127 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:20:36.388160 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:20:36.390868 (+ 2708us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:20:36.391494 (+ 626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:20:36.391501 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:20:36.391505 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:20:36.391656 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:20:36.392027 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:20:36.392032 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:20:36.392036 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:20:36.392313 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:20:36.392590 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0620 20:20:36.392868 (+ 278us) server_negotiation.cc:299] Negotiation successful
0620 20:20:36.393096 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":397,"thread_start_us":245,"threads_started":1}
I20260620 20:20:36.394322 1840 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:20:36.477535 17252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu with pid 1717
2026-06-20T20:20:36Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (64844 ms)
[----------] 30 tests from SecurityITest (299750 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986536687152-17252-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 20:20:36 dist-test-slave-2hpg krb5kdc[1910](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 20:20:36 dist-test-slave-2hpg krb5kdc[1910](info): set up 2 sockets
Jun 20 20:20:36 dist-test-slave-2hpg krb5kdc[1910](info): commencing operation
krb5kdc: starting...
W20260620 20:20:37.602375 17252 mini_kdc.cc:121] Time spent starting KDC: real 1.084s user 0.002s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986536687152-17252-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986536687152-17252-0/krb5kdc/test-user.keytab.
Jun 20 20:20:37 dist-test-slave-2hpg krb5kdc[1910](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986837, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T20:20:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T20:20:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.217.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.217.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.217.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.62.keytab.
Entry for principal kudu/127.16.217.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.62.keytab.
I20260620 20:20:37.847160 17252 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
/tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986536687152-17252-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986536687152-17252-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986536687152-17252-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986536687152-17252-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.217.62:45647
--webserver_interface=127.16.217.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.62.keytab
--principal=kudu/127.16.217.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.217.20:34051
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.217.62:45647
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986536687152-17252-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986536687152-17252-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986536687152-17252-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:20:38.174343 1923 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:20:38.174688 1923 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:20:38.174854 1923 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:20:38.182210 1923 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:20:38.182361 1923 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:20:38.182457 1923 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:20:38.182561 1923 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:20:38.182648 1923 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:20:38.192600 1923 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.217.20:34051
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986536687152-17252-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986536687152-17252-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.217.62:45647
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.62.keytab
--principal=kudu/127.16.217.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.217.62:45647
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986536687152-17252-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.217.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986536687152-17252-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:20:38.193950 1923 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:20:38.195282 1923 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:20:38.206650 1928 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:20:38.206693 1931 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:20:38.206717 1929 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:20:38.207899 1923 server_base.cc:1034] running on GCE node
I20260620 20:20:38.208618 1923 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:20:38.210040 1923 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:20:38.211256 1923 hybrid_clock.cc:648] HybridClock initialized: now 1781986838211197 us; error 53 us; skew 500 ppm
I20260620 20:20:38.211601 1923 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:20:38 dist-test-slave-2hpg krb5kdc[1910](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986838, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.217.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:20:38.214668 1923 init.cc:377] Logged in from keytab as kudu/127.16.217.62@KRBTEST.COM (short username kudu)
I20260620 20:20:38.216037 1923 webserver.cc:460] Webserver started at http://127.16.217.62:38073/ using document root <none> and password file <none>
I20260620 20:20:38.216653 1923 fs_manager.cc:359] Metadata directory not provided
I20260620 20:20:38.216754 1923 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:20:38.217007 1923 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:20:38.219926 1923 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986536687152-17252-0/minicluster-data/master-0/data/instance:
uuid: "859fecb705a1417fa625c6317c298836"
format_stamp: "Formatted at 2026-06-20 20:20:38 on dist-test-slave-2hpg"
server_key: "d1b65491e7b666419b87e570ff241a80"
server_key_iv: "95c4ab42d166d9669567f5d8bb5aa2a1"
server_key_version: "encryptionkey@0"
I20260620 20:20:38.220608 1923 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986536687152-17252-0/minicluster-data/master-0/wal/instance:
uuid: "859fecb705a1417fa625c6317c298836"
format_stamp: "Formatted at 2026-06-20 20:20:38 on dist-test-slave-2hpg"
server_key: "d1b65491e7b666419b87e570ff241a80"
server_key_iv: "95c4ab42d166d9669567f5d8bb5aa2a1"
server_key_version: "encryptionkey@0"
I20260620 20:20:38.225636 1923 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 20:20:38.229012 1938 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:20:38.230402 1923 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 20:20:38.230589 1923 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986536687152-17252-0/minicluster-data/master-0/data,/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986536687152-17252-0/minicluster-data/master-0/wal
uuid: "859fecb705a1417fa625c6317c298836"
format_stamp: "Formatted at 2026-06-20 20:20:38 on dist-test-slave-2hpg"
server_key: "d1b65491e7b666419b87e570ff241a80"
server_key_iv: "95c4ab42d166d9669567f5d8bb5aa2a1"
server_key_version: "encryptionkey@0"
I20260620 20:20:38.230790 1923 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986536687152-17252-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986536687152-17252-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986536687152-17252-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:20:38.247584 1923 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:20:38.251690 1923 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:20:38.251984 1923 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:20:38.271339 1923 rpc_server.cc:307] RPC server started. Bound to: 127.16.217.62:45647
I20260620 20:20:38.271360 1989 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.217.62:45647 every 8 connection(s)
I20260620 20:20:38.272826 1923 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986536687152-17252-0/minicluster-data/master-0/data/info.pb
I20260620 20:20:38.276646 1990 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:20:38.279394 17252 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu as pid 1923
I20260620 20:20:38.279662 17252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986536687152-17252-0/minicluster-data/master-0/wal/instance
I20260620 20:20:38.280321 17252 external_mini_cluster.cc:1428] Setting key fb9c7ebbcd9c4c6bb1adcf5ad50e30aa
I20260620 20:20:38.290347 1990 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 859fecb705a1417fa625c6317c298836: Bootstrap starting.
I20260620 20:20:38.294061 1990 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 859fecb705a1417fa625c6317c298836: Neither blocks nor log segments found. Creating new log.
I20260620 20:20:38.295130 1990 log.cc:824] T 00000000000000000000000000000000 P 859fecb705a1417fa625c6317c298836: Log is configured to *not* fsync() on all Append() calls
Jun 20 20:20:38 dist-test-slave-2hpg krb5kdc[1910](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986837, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.217.62@KRBTEST.COM
I20260620 20:20:38.298846 1990 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 859fecb705a1417fa625c6317c298836: No bootstrap required, opened a new log
I20260620 20:20:38.308137 1993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:38.282491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51990 (local address 127.16.217.62:45647)
0620 20:20:38.283286 (+ 795us) server_negotiation.cc:206] Beginning negotiation
0620 20:20:38.283302 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 20:20:38.283354 (+ 52us) server_negotiation.cc:407] Connection header received
0620 20:20:38.285947 (+ 2593us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:20:38.285980 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:20:38.286605 (+ 625us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:20:38.287085 (+ 480us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:20:38.288448 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:38.290496 (+ 2048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:38.291920 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:38.292978 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:38.293203 (+ 225us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:20:38.298682 (+ 5479us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:20:38.298694 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:20:38.298713 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:20:38.298763 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:20:38.303187 (+ 4424us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:20:38.303952 (+ 765us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:20:38.303961 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:20:38.303971 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:20:38.304190 (+ 219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:20:38.304570 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:20:38.304577 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:20:38.304580 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:20:38.305346 (+ 766us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:20:38.305674 (+ 328us) server_negotiation.cc:1035] Waiting for connection context
0620 20:20:38.306159 (+ 485us) server_negotiation.cc:299] Negotiation successful
0620 20:20:38.306667 (+ 508us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":558,"thread_start_us":336,"threads_started":1}
I20260620 20:20:38.310271 1990 raft_consensus.cc:348] T 00000000000000000000000000000000 P 859fecb705a1417fa625c6317c298836 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "859fecb705a1417fa625c6317c298836" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 45647 } }
I20260620 20:20:38.310628 1990 raft_consensus.cc:374] T 00000000000000000000000000000000 P 859fecb705a1417fa625c6317c298836 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:20:38.310757 1990 raft_consensus.cc:729] T 00000000000000000000000000000000 P 859fecb705a1417fa625c6317c298836 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 859fecb705a1417fa625c6317c298836, State: Initialized, Role: FOLLOWER
I20260620 20:20:38.311452 1990 consensus_queue.cc:260] T 00000000000000000000000000000000 P 859fecb705a1417fa625c6317c298836 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "859fecb705a1417fa625c6317c298836" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 45647 } }
I20260620 20:20:38.311724 1990 raft_consensus.cc:388] T 00000000000000000000000000000000 P 859fecb705a1417fa625c6317c298836 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 20:20:38.311856 1990 raft_consensus.cc:482] T 00000000000000000000000000000000 P 859fecb705a1417fa625c6317c298836 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 20:20:38.312033 1990 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 859fecb705a1417fa625c6317c298836 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:20:38.314661 1990 raft_consensus.cc:504] T 00000000000000000000000000000000 P 859fecb705a1417fa625c6317c298836 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "859fecb705a1417fa625c6317c298836" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 45647 } }
I20260620 20:20:38.315200 1990 leader_election.cc:302] T 00000000000000000000000000000000 P 859fecb705a1417fa625c6317c298836 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 859fecb705a1417fa625c6317c298836; no voters:
I20260620 20:20:38.315831 1990 leader_election.cc:288] T 00000000000000000000000000000000 P 859fecb705a1417fa625c6317c298836 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 20:20:38.315975 1995 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 859fecb705a1417fa625c6317c298836 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:20:38.316668 1995 raft_consensus.cc:686] T 00000000000000000000000000000000 P 859fecb705a1417fa625c6317c298836 [term 1 LEADER]: Becoming Leader. State: Replica: 859fecb705a1417fa625c6317c298836, State: Running, Role: LEADER
I20260620 20:20:38.317204 1995 consensus_queue.cc:237] T 00000000000000000000000000000000 P 859fecb705a1417fa625c6317c298836 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "859fecb705a1417fa625c6317c298836" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 45647 } }
I20260620 20:20:38.317992 1990 sys_catalog.cc:564] T 00000000000000000000000000000000 P 859fecb705a1417fa625c6317c298836 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:20:38.320061 1996 sys_catalog.cc:455] T 00000000000000000000000000000000 P 859fecb705a1417fa625c6317c298836 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "859fecb705a1417fa625c6317c298836" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "859fecb705a1417fa625c6317c298836" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 45647 } } }
I20260620 20:20:38.320499 1996 sys_catalog.cc:458] T 00000000000000000000000000000000 P 859fecb705a1417fa625c6317c298836 [sys.catalog]: This master's current role is: LEADER
I20260620 20:20:38.325767 1997 sys_catalog.cc:455] T 00000000000000000000000000000000 P 859fecb705a1417fa625c6317c298836 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 859fecb705a1417fa625c6317c298836. Latest consensus state: current_term: 1 leader_uuid: "859fecb705a1417fa625c6317c298836" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "859fecb705a1417fa625c6317c298836" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 45647 } } }
I20260620 20:20:38.326658 1997 sys_catalog.cc:458] T 00000000000000000000000000000000 P 859fecb705a1417fa625c6317c298836 [sys.catalog]: This master's current role is: LEADER
I20260620 20:20:38.327026 2001 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 20:20:38.333271 2001 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 20:20:38.342868 2001 catalog_manager.cc:1347] Generated new cluster ID: 8b5d0ea1ad1f4592813f479c89edefe4
I20260620 20:20:38.342994 2001 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 20:20:38.359149 2001 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 20:20:38.361086 2001 catalog_manager.cc:1504] Loading token signing keys...
I20260620 20:20:38.378057 2001 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 859fecb705a1417fa625c6317c298836: Generated new TSK 0
I20260620 20:20:38.378945 2001 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.217.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.217.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.217.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.1.keytab.
Entry for principal kudu/127.16.217.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.1.keytab.
I20260620 20:20:38.445582 17252 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
/tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986536687152-17252-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986536687152-17252-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986536687152-17252-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986536687152-17252-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.217.1:0
--local_ip_for_outbound_sockets=127.16.217.1
--webserver_interface=127.16.217.1
--webserver_port=0
--tserver_master_addrs=127.16.217.62:45647
--keytab_file=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.1.keytab
--principal=kudu/127.16.217.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.217.20:34051
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986536687152-17252-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986536687152-17252-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986536687152-17252-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:20:38.850651 2016 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:20:38.850942 2016 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:20:38.851133 2016 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:20:38.858289 2016 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:20:38.858431 2016 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:20:38.858608 2016 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.217.1
I20260620 20:20:38.869757 2016 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.217.20:34051
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986536687152-17252-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986536687152-17252-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.1.keytab
--principal=kudu/127.16.217.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.217.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986536687152-17252-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.217.1
--webserver_port=0
--tserver_master_addrs=127.16.217.62:45647
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.217.1
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986536687152-17252-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:20:38.871098 2016 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:20:38.872421 2016 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:20:38.888527 2022 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:20:38.889127 2021 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:20:38.889573 2016 server_base.cc:1034] running on GCE node
W20260620 20:20:38.889721 2024 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:20:38.890395 2016 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:20:38.891420 2016 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:20:38.892639 2016 hybrid_clock.cc:648] HybridClock initialized: now 1781986838892572 us; error 69 us; skew 500 ppm
I20260620 20:20:38.892954 2016 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:20:38 dist-test-slave-2hpg krb5kdc[1910](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986838, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.217.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:20:38.896301 2016 init.cc:377] Logged in from keytab as kudu/127.16.217.1@KRBTEST.COM (short username kudu)
I20260620 20:20:38.897925 2016 webserver.cc:460] Webserver started at http://127.16.217.1:42375/ using document root <none> and password file <none>
I20260620 20:20:38.898617 2016 fs_manager.cc:359] Metadata directory not provided
I20260620 20:20:38.898762 2016 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:20:38.899073 2016 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:20:38.902004 2016 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986536687152-17252-0/minicluster-data/ts-0/data/instance:
uuid: "19a96e445738489db15cbc247b2ad210"
format_stamp: "Formatted at 2026-06-20 20:20:38 on dist-test-slave-2hpg"
server_key: "1a2c0be7267536cf0fbe52e83d7b5650"
server_key_iv: "7a22cb699b7f35de5d082363bd07032b"
server_key_version: "encryptionkey@0"
I20260620 20:20:38.902752 2016 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986536687152-17252-0/minicluster-data/ts-0/wal/instance:
uuid: "19a96e445738489db15cbc247b2ad210"
format_stamp: "Formatted at 2026-06-20 20:20:38 on dist-test-slave-2hpg"
server_key: "1a2c0be7267536cf0fbe52e83d7b5650"
server_key_iv: "7a22cb699b7f35de5d082363bd07032b"
server_key_version: "encryptionkey@0"
I20260620 20:20:38.908092 2016 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.005s
I20260620 20:20:38.911832 2031 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:20:38.913158 2016 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 20:20:38.913340 2016 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986536687152-17252-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986536687152-17252-0/minicluster-data/ts-0/wal
uuid: "19a96e445738489db15cbc247b2ad210"
format_stamp: "Formatted at 2026-06-20 20:20:38 on dist-test-slave-2hpg"
server_key: "1a2c0be7267536cf0fbe52e83d7b5650"
server_key_iv: "7a22cb699b7f35de5d082363bd07032b"
server_key_version: "encryptionkey@0"
I20260620 20:20:38.913571 2016 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986536687152-17252-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986536687152-17252-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986536687152-17252-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:20:38.929442 2016 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:20:38.933604 2016 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:20:38.934034 2016 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:20:38.935070 2016 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:20:38.936483 2016 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:20:38.936573 2016 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:20:38.936663 2016 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:20:38.936720 2016 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:20:38.971963 2016 rpc_server.cc:307] RPC server started. Bound to: 127.16.217.1:43375
I20260620 20:20:38.972023 2143 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.217.1:43375 every 8 connection(s)
I20260620 20:20:38.973495 2016 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986536687152-17252-0/minicluster-data/ts-0/data/info.pb
I20260620 20:20:38.983534 17252 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu as pid 2016
I20260620 20:20:38.983774 17252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986536687152-17252-0/minicluster-data/ts-0/wal/instance
I20260620 20:20:38.984443 17252 external_mini_cluster.cc:1428] Setting key 300621cd0c5f1ce5259478c217517c7a
Jun 20 20:20:38 dist-test-slave-2hpg krb5kdc[1910](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986838, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.217.1@KRBTEST.COM for kudu/127.16.217.62@KRBTEST.COM
I20260620 20:20:38.994238 2146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:38.976223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.217.1:37465 (local address 127.16.217.62:45647)
0620 20:20:38.976710 (+ 487us) server_negotiation.cc:206] Beginning negotiation
0620 20:20:38.976714 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:20:38.977616 (+ 902us) server_negotiation.cc:407] Connection header received
0620 20:20:38.979917 (+ 2301us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:20:38.979922 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:20:38.980018 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:20:38.980142 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:20:38.981674 (+ 1532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:38.982445 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:38.984101 (+ 1656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:38.984495 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:38.984633 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:20:38.988565 (+ 3932us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:20:38.988599 (+ 34us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:20:38.988603 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:20:38.988634 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:20:38.991257 (+ 2623us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:20:38.991942 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:20:38.991949 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:20:38.991953 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:20:38.992078 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:20:38.992497 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:20:38.992503 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:20:38.992507 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:20:38.992780 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:20:38.992935 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0620 20:20:38.993635 (+ 700us) server_negotiation.cc:299] Negotiation successful
0620 20:20:38.993853 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":378,"thread_start_us":244,"threads_started":1}
I20260620 20:20:38.995585 2147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:38.976654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:45647 (local address 127.16.217.1:37465)
0620 20:20:38.977417 (+ 763us) negotiation.cc:107] Waiting for socket to connect
0620 20:20:38.977459 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 20:20:38.979621 (+ 2162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:20:38.980387 (+ 766us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:20:38.980396 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:20:38.980914 (+ 518us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:20:38.981490 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:20:38.981515 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:20:38.982617 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:20:38.982622 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:20:38.983957 (+ 1335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:20:38.983969 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:20:38.984748 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:20:38.984757 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:20:38.985075 (+ 318us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:20:38.985860 (+ 785us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:20:38.985948 (+ 88us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:20:38.988345 (+ 2397us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:20:38.991395 (+ 3050us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:20:38.991403 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:20:38.991419 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:20:38.991824 (+ 405us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:20:38.992187 (+ 363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:20:38.992192 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:20:38.992196 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:20:38.992385 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:20:38.992919 (+ 534us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:20:38.992926 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:20:38.993254 (+ 328us) client_negotiation.cc:769] Sending connection context
0620 20:20:38.993514 (+ 260us) client_negotiation.cc:240] Negotiation successful
0620 20:20:38.994053 (+ 539us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":528,"thread_start_us":336,"threads_started":1}
I20260620 20:20:38.997212 2144 heartbeater.cc:347] Connected to a master server at 127.16.217.62:45647
I20260620 20:20:38.997550 2144 heartbeater.cc:464] Registering TS with master...
I20260620 20:20:38.998399 2144 heartbeater.cc:511] Master 127.16.217.62:45647 requested a full tablet report, sending...
I20260620 20:20:39.000675 1955 ts_manager.cc:194] Registered new tserver with Master: 19a96e445738489db15cbc247b2ad210 (127.16.217.1:43375)
I20260620 20:20:39.002426 1955 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.217.1@KRBTEST.COM'} at 127.16.217.1:37465
WARNING: no policy specified for kudu/127.16.217.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.217.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.217.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.2.keytab.
Entry for principal kudu/127.16.217.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.2.keytab.
I20260620 20:20:39.041174 17252 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
/tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986536687152-17252-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986536687152-17252-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986536687152-17252-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986536687152-17252-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.217.2:0
--local_ip_for_outbound_sockets=127.16.217.2
--webserver_interface=127.16.217.2
--webserver_port=0
--tserver_master_addrs=127.16.217.62:45647
--keytab_file=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.2.keytab
--principal=kudu/127.16.217.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.217.20:34051
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986536687152-17252-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986536687152-17252-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986536687152-17252-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:20:39.382094 2150 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:20:39.382347 2150 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:20:39.382550 2150 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:20:39.389565 2150 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:20:39.389660 2150 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:20:39.389891 2150 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.217.2
I20260620 20:20:39.399355 2150 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.217.20:34051
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986536687152-17252-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986536687152-17252-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.2.keytab
--principal=kudu/127.16.217.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.217.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986536687152-17252-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.217.2
--webserver_port=0
--tserver_master_addrs=127.16.217.62:45647
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.217.2
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986536687152-17252-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:20:39.400625 2150 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:20:39.401995 2150 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:20:39.414747 2156 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:20:39.414805 2155 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:20:39.414783 2158 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:20:39.415871 2150 server_base.cc:1034] running on GCE node
I20260620 20:20:39.416455 2150 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:20:39.417464 2150 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:20:39.418668 2150 hybrid_clock.cc:648] HybridClock initialized: now 1781986839418602 us; error 53 us; skew 500 ppm
I20260620 20:20:39.419023 2150 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:20:39 dist-test-slave-2hpg krb5kdc[1910](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986839, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.217.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:20:39.422169 2150 init.cc:377] Logged in from keytab as kudu/127.16.217.2@KRBTEST.COM (short username kudu)
I20260620 20:20:39.423624 2150 webserver.cc:460] Webserver started at http://127.16.217.2:36619/ using document root <none> and password file <none>
I20260620 20:20:39.424283 2150 fs_manager.cc:359] Metadata directory not provided
I20260620 20:20:39.424430 2150 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:20:39.424757 2150 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:20:39.427788 2150 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986536687152-17252-0/minicluster-data/ts-1/data/instance:
uuid: "53ac50f050ee4a3681e16f45546e4b72"
format_stamp: "Formatted at 2026-06-20 20:20:39 on dist-test-slave-2hpg"
server_key: "b5067b3bda29c0850fbe1ef85ded7fde"
server_key_iv: "ecf6bf550aca7069cc51040b3ceb37b8"
server_key_version: "encryptionkey@0"
I20260620 20:20:39.428527 2150 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986536687152-17252-0/minicluster-data/ts-1/wal/instance:
uuid: "53ac50f050ee4a3681e16f45546e4b72"
format_stamp: "Formatted at 2026-06-20 20:20:39 on dist-test-slave-2hpg"
server_key: "b5067b3bda29c0850fbe1ef85ded7fde"
server_key_iv: "ecf6bf550aca7069cc51040b3ceb37b8"
server_key_version: "encryptionkey@0"
I20260620 20:20:39.433609 2150 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260620 20:20:39.436970 2165 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:20:39.438238 2150 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 20:20:39.438408 2150 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986536687152-17252-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986536687152-17252-0/minicluster-data/ts-1/wal
uuid: "53ac50f050ee4a3681e16f45546e4b72"
format_stamp: "Formatted at 2026-06-20 20:20:39 on dist-test-slave-2hpg"
server_key: "b5067b3bda29c0850fbe1ef85ded7fde"
server_key_iv: "ecf6bf550aca7069cc51040b3ceb37b8"
server_key_version: "encryptionkey@0"
I20260620 20:20:39.438612 2150 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986536687152-17252-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986536687152-17252-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986536687152-17252-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:20:39.469012 2150 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:20:39.473268 2150 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:20:39.473574 2150 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:20:39.474709 2150 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:20:39.476198 2150 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:20:39.476301 2150 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:20:39.476459 2150 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:20:39.476537 2150 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:20:39.510816 2150 rpc_server.cc:307] RPC server started. Bound to: 127.16.217.2:43763
I20260620 20:20:39.510905 2277 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.217.2:43763 every 8 connection(s)
I20260620 20:20:39.512137 2150 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986536687152-17252-0/minicluster-data/ts-1/data/info.pb
I20260620 20:20:39.514773 17252 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu as pid 2150
I20260620 20:20:39.514973 17252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986536687152-17252-0/minicluster-data/ts-1/wal/instance
I20260620 20:20:39.515519 17252 external_mini_cluster.cc:1428] Setting key 9f2c5111f003eaaf259434d277c755f4
Jun 20 20:20:39 dist-test-slave-2hpg krb5kdc[1910](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986839, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.217.2@KRBTEST.COM for kudu/127.16.217.62@KRBTEST.COM
I20260620 20:20:39.536764 2280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:39.514863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.217.2:52999 (local address 127.16.217.62:45647)
0620 20:20:39.517173 (+ 2310us) server_negotiation.cc:206] Beginning negotiation
0620 20:20:39.517179 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:20:39.517200 (+ 21us) server_negotiation.cc:407] Connection header received
0620 20:20:39.519693 (+ 2493us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:20:39.519698 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:20:39.519774 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:20:39.519881 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:20:39.522041 (+ 2160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:39.522764 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:39.525253 (+ 2489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:39.525719 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:39.525816 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:20:39.529620 (+ 3804us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:20:39.529629 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:20:39.529637 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:20:39.529682 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:20:39.533334 (+ 3652us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:20:39.534154 (+ 820us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:20:39.534162 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:20:39.534168 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:20:39.534322 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:20:39.534756 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:20:39.534765 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:20:39.534771 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:20:39.535172 (+ 401us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:20:39.535327 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0620 20:20:39.536017 (+ 690us) server_negotiation.cc:299] Negotiation successful
0620 20:20:39.536258 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2201,"thread_start_us":2091,"threads_started":1}
I20260620 20:20:39.537307 2281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:39.515565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:45647 (local address 127.16.217.2:52999)
0620 20:20:39.516639 (+ 1074us) negotiation.cc:107] Waiting for socket to connect
0620 20:20:39.516700 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0620 20:20:39.519386 (+ 2686us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:20:39.520168 (+ 782us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:20:39.520181 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:20:39.520973 (+ 792us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:20:39.521817 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:20:39.521844 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:20:39.522929 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:20:39.522937 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:20:39.525094 (+ 2157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:20:39.525113 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:20:39.525973 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:20:39.525983 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:20:39.526343 (+ 360us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:20:39.527076 (+ 733us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:20:39.527112 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:20:39.529391 (+ 2279us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:20:39.533475 (+ 4084us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:20:39.533481 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:20:39.533493 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:20:39.534028 (+ 535us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:20:39.534437 (+ 409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:20:39.534440 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:20:39.534443 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:20:39.534659 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:20:39.535329 (+ 670us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:20:39.535335 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:20:39.535646 (+ 311us) client_negotiation.cc:769] Sending connection context
0620 20:20:39.535863 (+ 217us) client_negotiation.cc:240] Negotiation successful
0620 20:20:39.536218 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":678,"thread_start_us":364,"threads_started":1}
I20260620 20:20:39.538847 2278 heartbeater.cc:347] Connected to a master server at 127.16.217.62:45647
I20260620 20:20:39.539167 2278 heartbeater.cc:464] Registering TS with master...
I20260620 20:20:39.539932 2278 heartbeater.cc:511] Master 127.16.217.62:45647 requested a full tablet report, sending...
I20260620 20:20:39.541481 1955 ts_manager.cc:194] Registered new tserver with Master: 53ac50f050ee4a3681e16f45546e4b72 (127.16.217.2:43763)
I20260620 20:20:39.542508 1955 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.217.2@KRBTEST.COM'} at 127.16.217.2:52999
WARNING: no policy specified for kudu/127.16.217.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.217.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.217.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.3.keytab.
Entry for principal kudu/127.16.217.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.3.keytab.
I20260620 20:20:39.573622 17252 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
/tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986536687152-17252-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986536687152-17252-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986536687152-17252-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986536687152-17252-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.217.3:0
--local_ip_for_outbound_sockets=127.16.217.3
--webserver_interface=127.16.217.3
--webserver_port=0
--tserver_master_addrs=127.16.217.62:45647
--keytab_file=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.3.keytab
--principal=kudu/127.16.217.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.217.20:34051
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986536687152-17252-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986536687152-17252-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986536687152-17252-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:20:39.909754 2284 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:20:39.910102 2284 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:20:39.910208 2284 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:20:39.917639 2284 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:20:39.917740 2284 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:20:39.917852 2284 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.217.3
I20260620 20:20:39.928123 2284 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.217.20:34051
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986536687152-17252-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986536687152-17252-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.3.keytab
--principal=kudu/127.16.217.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.217.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986536687152-17252-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.217.3
--webserver_port=0
--tserver_master_addrs=127.16.217.62:45647
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.217.3
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986536687152-17252-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:20:39.929340 2284 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:20:39.930616 2284 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:20:39.943815 2289 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:20:39.943789 2290 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:20:39.944308 2292 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:20:39.945278 2284 server_base.cc:1034] running on GCE node
I20260620 20:20:39.946007 2284 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:20:39.947029 2284 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:20:39.948262 2284 hybrid_clock.cc:648] HybridClock initialized: now 1781986839948218 us; error 47 us; skew 500 ppm
I20260620 20:20:39.948586 2284 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:20:39 dist-test-slave-2hpg krb5kdc[1910](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986839, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.217.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:20:39.951707 2284 init.cc:377] Logged in from keytab as kudu/127.16.217.3@KRBTEST.COM (short username kudu)
I20260620 20:20:39.953104 2284 webserver.cc:460] Webserver started at http://127.16.217.3:34075/ using document root <none> and password file <none>
I20260620 20:20:39.953711 2284 fs_manager.cc:359] Metadata directory not provided
I20260620 20:20:39.953804 2284 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:20:39.954134 2284 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:20:39.956979 2284 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986536687152-17252-0/minicluster-data/ts-2/data/instance:
uuid: "46215a0dc4cc40e48eafa1ac007e1e06"
format_stamp: "Formatted at 2026-06-20 20:20:39 on dist-test-slave-2hpg"
server_key: "b755484d069ed8cc098280770233eabe"
server_key_iv: "8969ef6c513ee4aaf107e25248849511"
server_key_version: "encryptionkey@0"
I20260620 20:20:39.957674 2284 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986536687152-17252-0/minicluster-data/ts-2/wal/instance:
uuid: "46215a0dc4cc40e48eafa1ac007e1e06"
format_stamp: "Formatted at 2026-06-20 20:20:39 on dist-test-slave-2hpg"
server_key: "b755484d069ed8cc098280770233eabe"
server_key_iv: "8969ef6c513ee4aaf107e25248849511"
server_key_version: "encryptionkey@0"
I20260620 20:20:39.962579 2284 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 20:20:39.965813 2299 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:20:39.967254 2284 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 20:20:39.967417 2284 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986536687152-17252-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986536687152-17252-0/minicluster-data/ts-2/wal
uuid: "46215a0dc4cc40e48eafa1ac007e1e06"
format_stamp: "Formatted at 2026-06-20 20:20:39 on dist-test-slave-2hpg"
server_key: "b755484d069ed8cc098280770233eabe"
server_key_iv: "8969ef6c513ee4aaf107e25248849511"
server_key_version: "encryptionkey@0"
I20260620 20:20:39.967613 2284 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986536687152-17252-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986536687152-17252-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986536687152-17252-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:20:39.991671 2284 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:20:39.995880 2284 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:20:39.996176 2284 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:20:39.997205 2284 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:20:39.998606 2284 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:20:39.998677 2284 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:20:39.998829 2284 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:20:39.998881 2284 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:20:40.005785 2144 heartbeater.cc:503] Master 127.16.217.62:45647 was elected leader, sending a full tablet report...
I20260620 20:20:40.034206 2284 rpc_server.cc:307] RPC server started. Bound to: 127.16.217.3:41347
I20260620 20:20:40.034301 2411 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.217.3:41347 every 8 connection(s)
I20260620 20:20:40.035634 2284 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986536687152-17252-0/minicluster-data/ts-2/data/info.pb
I20260620 20:20:40.038401 17252 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu as pid 2284
I20260620 20:20:40.038580 17252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986536687152-17252-0/minicluster-data/ts-2/wal/instance
I20260620 20:20:40.039152 17252 external_mini_cluster.cc:1428] Setting key 9d7f62672cb4f2e623a8aa5d2819c094
Jun 20 20:20:40 dist-test-slave-2hpg krb5kdc[1910](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986839, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.217.3@KRBTEST.COM for kudu/127.16.217.62@KRBTEST.COM
I20260620 20:20:40.056070 2414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:40.038151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.217.3:52249 (local address 127.16.217.62:45647)
0620 20:20:40.038701 (+ 550us) server_negotiation.cc:206] Beginning negotiation
0620 20:20:40.038710 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 20:20:40.039848 (+ 1138us) server_negotiation.cc:407] Connection header received
0620 20:20:40.041687 (+ 1839us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:20:40.041692 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:20:40.041788 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:20:40.041944 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:20:40.043480 (+ 1536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:40.044253 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:40.045761 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:40.046219 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:40.046353 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:20:40.049672 (+ 3319us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:20:40.049678 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:20:40.049682 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:20:40.049722 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:20:40.052726 (+ 3004us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:20:40.053487 (+ 761us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:20:40.053494 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:20:40.053498 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:20:40.053655 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:20:40.054248 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:20:40.054254 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:20:40.054259 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:20:40.054713 (+ 454us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:20:40.054876 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0620 20:20:40.055561 (+ 685us) server_negotiation.cc:299] Negotiation successful
0620 20:20:40.055740 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":399,"thread_start_us":236,"threads_started":1}
I20260620 20:20:40.056895 2415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:40.038913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:45647 (local address 127.16.217.3:52249)
0620 20:20:40.039701 (+ 788us) negotiation.cc:107] Waiting for socket to connect
0620 20:20:40.039738 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 20:20:40.041406 (+ 1668us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:20:40.042170 (+ 764us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:20:40.042182 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:20:40.042764 (+ 582us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:20:40.043320 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:20:40.043336 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:20:40.044408 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:20:40.044412 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:20:40.045615 (+ 1203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:20:40.045626 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:20:40.046433 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:20:40.046441 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:20:40.046672 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:20:40.047270 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:20:40.047290 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:20:40.049442 (+ 2152us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:20:40.052872 (+ 3430us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:20:40.052883 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:20:40.052898 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:20:40.053339 (+ 441us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:20:40.053798 (+ 459us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:20:40.053802 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:20:40.053804 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:20:40.054098 (+ 294us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:20:40.054884 (+ 786us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:20:40.054891 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:20:40.055220 (+ 329us) client_negotiation.cc:769] Sending connection context
0620 20:20:40.055443 (+ 223us) client_negotiation.cc:240] Negotiation successful
0620 20:20:40.055809 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":547,"thread_start_us":348,"threads_started":1}
I20260620 20:20:40.058362 2412 heartbeater.cc:347] Connected to a master server at 127.16.217.62:45647
I20260620 20:20:40.058681 2412 heartbeater.cc:464] Registering TS with master...
I20260620 20:20:40.059381 2412 heartbeater.cc:511] Master 127.16.217.62:45647 requested a full tablet report, sending...
I20260620 20:20:40.060760 1955 ts_manager.cc:194] Registered new tserver with Master: 46215a0dc4cc40e48eafa1ac007e1e06 (127.16.217.3:41347)
I20260620 20:20:40.061582 1955 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.217.3@KRBTEST.COM'} at 127.16.217.3:52249
I20260620 20:20:40.067329 17252 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 20:20:40.085462 2414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:40.071978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52000 (local address 127.16.217.62:45647)
0620 20:20:40.072149 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0620 20:20:40.072153 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:20:40.072618 (+ 465us) server_negotiation.cc:407] Connection header received
0620 20:20:40.072748 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:20:40.072751 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:20:40.072822 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:20:40.072903 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:20:40.074377 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:40.075112 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:40.076586 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:40.076996 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:40.077106 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:20:40.079484 (+ 2378us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:20:40.079489 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:20:40.079493 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:20:40.079578 (+ 85us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:20:40.082956 (+ 3378us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:20:40.083630 (+ 674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:20:40.083635 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:20:40.083638 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:20:40.083772 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:20:40.084107 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:20:40.084111 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:20:40.084114 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:20:40.084463 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:20:40.084677 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0620 20:20:40.085029 (+ 352us) server_negotiation.cc:299] Negotiation successful
0620 20:20:40.085219 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 20:20:40.089756 1955 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52000:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 20:20:40.097745 1955 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 20:20:40.131634 2430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:40.112675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60480 (local address 127.16.217.2:43763)
0620 20:20:40.119600 (+ 6925us) server_negotiation.cc:206] Beginning negotiation
0620 20:20:40.119613 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 20:20:40.123831 (+ 4218us) server_negotiation.cc:407] Connection header received
0620 20:20:40.123978 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:20:40.123987 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:20:40.124206 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:20:40.124395 (+ 189us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:20:40.125304 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:40.126832 (+ 1528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:40.129126 (+ 2294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:40.130342 (+ 1216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:40.130478 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:20:40.130678 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0620 20:20:40.131088 (+ 410us) server_negotiation.cc:299] Negotiation successful
0620 20:20:40.131276 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6161,"thread_start_us":6016,"threads_started":1}
I20260620 20:20:40.131747 2425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:40.112070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.2:43763 (local address 127.0.0.1:60480)
0620 20:20:40.123705 (+ 11635us) negotiation.cc:107] Waiting for socket to connect
0620 20:20:40.123721 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 20:20:40.123836 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:20:40.124398 (+ 562us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:20:40.124402 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:20:40.124455 (+ 53us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:20:40.125061 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:20:40.125081 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:20:40.126996 (+ 1915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:20:40.127001 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:20:40.128945 (+ 1944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:20:40.128976 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:20:40.130609 (+ 1633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:20:40.130616 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:20:40.130817 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:20:40.130866 (+ 49us) client_negotiation.cc:769] Sending connection context
0620 20:20:40.131011 (+ 145us) client_negotiation.cc:240] Negotiation successful
0620 20:20:40.131199 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11530,"mutex_wait_us":133,"spinlock_wait_cycles":4864,"thread_start_us":10647,"threads_started":1}
I20260620 20:20:40.132702 2428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:40.111086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57222 (local address 127.16.217.3:41347)
0620 20:20:40.122694 (+ 11608us) server_negotiation.cc:206] Beginning negotiation
0620 20:20:40.122704 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 20:20:40.123448 (+ 744us) server_negotiation.cc:407] Connection header received
0620 20:20:40.123582 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:20:40.123591 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:20:40.123871 (+ 280us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:20:40.124074 (+ 203us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:20:40.125211 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:40.126537 (+ 1326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:40.129854 (+ 3317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:40.131150 (+ 1296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:40.131273 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:20:40.131481 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0620 20:20:40.132192 (+ 711us) server_negotiation.cc:299] Negotiation successful
0620 20:20:40.132372 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11425,"thread_start_us":11264,"threads_started":1}
I20260620 20:20:40.132829 2427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:40.109943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.1:43375 (local address 127.0.0.1:38228)
0620 20:20:40.111757 (+ 1814us) negotiation.cc:107] Waiting for socket to connect
0620 20:20:40.111807 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0620 20:20:40.112041 (+ 234us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:20:40.124116 (+ 12075us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:20:40.124123 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:20:40.124151 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:20:40.124641 (+ 490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:20:40.124658 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:20:40.126755 (+ 2097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:20:40.126762 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:20:40.129686 (+ 2924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:20:40.129701 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:20:40.131882 (+ 2181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:20:40.131890 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:20:40.132168 (+ 278us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:20:40.132206 (+ 38us) client_negotiation.cc:769] Sending connection context
0620 20:20:40.132294 (+ 88us) client_negotiation.cc:240] Negotiation successful
0620 20:20:40.132422 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1549,"mutex_wait_us":229,"thread_start_us":13522,"threads_started":1}
I20260620 20:20:40.133394 2429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:40.110290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.3:41347 (local address 127.0.0.1:57222)
0620 20:20:40.123081 (+ 12791us) negotiation.cc:107] Waiting for socket to connect
0620 20:20:40.123106 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0620 20:20:40.123277 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:20:40.124394 (+ 1117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:20:40.124401 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:20:40.124433 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:20:40.125032 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:20:40.125043 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:20:40.126700 (+ 1657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:20:40.126706 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:20:40.129686 (+ 2980us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:20:40.129701 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:20:40.131657 (+ 1956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:20:40.131666 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:20:40.131877 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:20:40.131964 (+ 87us) client_negotiation.cc:769] Sending connection context
0620 20:20:40.132065 (+ 101us) client_negotiation.cc:240] Negotiation successful
0620 20:20:40.133076 (+ 1011us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12622,"mutex_wait_us":1038,"spinlock_wait_cycles":2816,"thread_start_us":1132,"threads_started":1}
I20260620 20:20:40.134248 2426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:40.110231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38228 (local address 127.16.217.1:43375)
0620 20:20:40.123464 (+ 13233us) server_negotiation.cc:206] Beginning negotiation
0620 20:20:40.123475 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 20:20:40.123508 (+ 33us) server_negotiation.cc:407] Connection header received
0620 20:20:40.123622 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:20:40.123630 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:20:40.123882 (+ 252us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:20:40.124100 (+ 218us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:20:40.124860 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:40.126595 (+ 1735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:40.129989 (+ 3394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:40.131297 (+ 1308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:40.131502 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:20:40.131732 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0620 20:20:40.133700 (+ 1968us) server_negotiation.cc:299] Negotiation successful
0620 20:20:40.133908 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13053,"thread_start_us":12889,"threads_started":1}
I20260620 20:20:40.136054 2213 tablet_service.cc:1467] Processing CreateTablet for tablet 1716ac367ee64917bb929aebb8ee9126 (DEFAULT_TABLE table=test-table [id=4d9c4343641f44be8ab32001654affeb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:20:40.137243 2213 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1716ac367ee64917bb929aebb8ee9126. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:20:40.136919 2347 tablet_service.cc:1467] Processing CreateTablet for tablet 1716ac367ee64917bb929aebb8ee9126 (DEFAULT_TABLE table=test-table [id=4d9c4343641f44be8ab32001654affeb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:20:40.137531 2079 tablet_service.cc:1467] Processing CreateTablet for tablet 1716ac367ee64917bb929aebb8ee9126 (DEFAULT_TABLE table=test-table [id=4d9c4343641f44be8ab32001654affeb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:20:40.138479 2347 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1716ac367ee64917bb929aebb8ee9126. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:20:40.138909 2079 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1716ac367ee64917bb929aebb8ee9126. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:20:40.150207 2431 tablet_bootstrap.cc:492] T 1716ac367ee64917bb929aebb8ee9126 P 53ac50f050ee4a3681e16f45546e4b72: Bootstrap starting.
I20260620 20:20:40.153496 2431 tablet_bootstrap.cc:654] T 1716ac367ee64917bb929aebb8ee9126 P 53ac50f050ee4a3681e16f45546e4b72: Neither blocks nor log segments found. Creating new log.
I20260620 20:20:40.154801 2431 log.cc:824] T 1716ac367ee64917bb929aebb8ee9126 P 53ac50f050ee4a3681e16f45546e4b72: Log is configured to *not* fsync() on all Append() calls
I20260620 20:20:40.155714 2432 tablet_bootstrap.cc:492] T 1716ac367ee64917bb929aebb8ee9126 P 19a96e445738489db15cbc247b2ad210: Bootstrap starting.
I20260620 20:20:40.156090 2433 tablet_bootstrap.cc:492] T 1716ac367ee64917bb929aebb8ee9126 P 46215a0dc4cc40e48eafa1ac007e1e06: Bootstrap starting.
I20260620 20:20:40.160625 2433 tablet_bootstrap.cc:654] T 1716ac367ee64917bb929aebb8ee9126 P 46215a0dc4cc40e48eafa1ac007e1e06: Neither blocks nor log segments found. Creating new log.
I20260620 20:20:40.161403 2432 tablet_bootstrap.cc:654] T 1716ac367ee64917bb929aebb8ee9126 P 19a96e445738489db15cbc247b2ad210: Neither blocks nor log segments found. Creating new log.
I20260620 20:20:40.162225 2433 log.cc:824] T 1716ac367ee64917bb929aebb8ee9126 P 46215a0dc4cc40e48eafa1ac007e1e06: Log is configured to *not* fsync() on all Append() calls
I20260620 20:20:40.162846 2432 log.cc:824] T 1716ac367ee64917bb929aebb8ee9126 P 19a96e445738489db15cbc247b2ad210: Log is configured to *not* fsync() on all Append() calls
I20260620 20:20:40.164347 2431 tablet_bootstrap.cc:492] T 1716ac367ee64917bb929aebb8ee9126 P 53ac50f050ee4a3681e16f45546e4b72: No bootstrap required, opened a new log
I20260620 20:20:40.164667 2431 ts_tablet_manager.cc:1397] T 1716ac367ee64917bb929aebb8ee9126 P 53ac50f050ee4a3681e16f45546e4b72: Time spent bootstrapping tablet: real 0.015s user 0.007s sys 0.007s
I20260620 20:20:40.166115 2432 tablet_bootstrap.cc:492] T 1716ac367ee64917bb929aebb8ee9126 P 19a96e445738489db15cbc247b2ad210: No bootstrap required, opened a new log
I20260620 20:20:40.166420 2432 ts_tablet_manager.cc:1397] T 1716ac367ee64917bb929aebb8ee9126 P 19a96e445738489db15cbc247b2ad210: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.006s
I20260620 20:20:40.167129 2433 tablet_bootstrap.cc:492] T 1716ac367ee64917bb929aebb8ee9126 P 46215a0dc4cc40e48eafa1ac007e1e06: No bootstrap required, opened a new log
I20260620 20:20:40.167419 2433 ts_tablet_manager.cc:1397] T 1716ac367ee64917bb929aebb8ee9126 P 46215a0dc4cc40e48eafa1ac007e1e06: Time spent bootstrapping tablet: real 0.013s user 0.007s sys 0.001s
I20260620 20:20:40.172616 2431 raft_consensus.cc:348] T 1716ac367ee64917bb929aebb8ee9126 P 53ac50f050ee4a3681e16f45546e4b72 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19a96e445738489db15cbc247b2ad210" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 43375 } } peers { permanent_uuid: "46215a0dc4cc40e48eafa1ac007e1e06" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41347 } } peers { permanent_uuid: "53ac50f050ee4a3681e16f45546e4b72" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 43763 } }
I20260620 20:20:40.173079 2431 raft_consensus.cc:374] T 1716ac367ee64917bb929aebb8ee9126 P 53ac50f050ee4a3681e16f45546e4b72 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:20:40.173179 2431 raft_consensus.cc:729] T 1716ac367ee64917bb929aebb8ee9126 P 53ac50f050ee4a3681e16f45546e4b72 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53ac50f050ee4a3681e16f45546e4b72, State: Initialized, Role: FOLLOWER
I20260620 20:20:40.173916 2431 consensus_queue.cc:260] T 1716ac367ee64917bb929aebb8ee9126 P 53ac50f050ee4a3681e16f45546e4b72 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19a96e445738489db15cbc247b2ad210" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 43375 } } peers { permanent_uuid: "46215a0dc4cc40e48eafa1ac007e1e06" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41347 } } peers { permanent_uuid: "53ac50f050ee4a3681e16f45546e4b72" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 43763 } }
I20260620 20:20:40.175104 2278 heartbeater.cc:503] Master 127.16.217.62:45647 was elected leader, sending a full tablet report...
I20260620 20:20:40.175587 2431 ts_tablet_manager.cc:1428] T 1716ac367ee64917bb929aebb8ee9126 P 53ac50f050ee4a3681e16f45546e4b72: Time spent starting tablet: real 0.011s user 0.003s sys 0.007s
I20260620 20:20:40.178350 2432 raft_consensus.cc:348] T 1716ac367ee64917bb929aebb8ee9126 P 19a96e445738489db15cbc247b2ad210 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19a96e445738489db15cbc247b2ad210" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 43375 } } peers { permanent_uuid: "46215a0dc4cc40e48eafa1ac007e1e06" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41347 } } peers { permanent_uuid: "53ac50f050ee4a3681e16f45546e4b72" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 43763 } }
I20260620 20:20:40.178757 2432 raft_consensus.cc:374] T 1716ac367ee64917bb929aebb8ee9126 P 19a96e445738489db15cbc247b2ad210 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:20:40.178895 2432 raft_consensus.cc:729] T 1716ac367ee64917bb929aebb8ee9126 P 19a96e445738489db15cbc247b2ad210 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19a96e445738489db15cbc247b2ad210, State: Initialized, Role: FOLLOWER
I20260620 20:20:40.179569 2432 consensus_queue.cc:260] T 1716ac367ee64917bb929aebb8ee9126 P 19a96e445738489db15cbc247b2ad210 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19a96e445738489db15cbc247b2ad210" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 43375 } } peers { permanent_uuid: "46215a0dc4cc40e48eafa1ac007e1e06" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41347 } } peers { permanent_uuid: "53ac50f050ee4a3681e16f45546e4b72" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 43763 } }
I20260620 20:20:40.181448 2432 ts_tablet_manager.cc:1428] T 1716ac367ee64917bb929aebb8ee9126 P 19a96e445738489db15cbc247b2ad210: Time spent starting tablet: real 0.015s user 0.014s sys 0.000s
I20260620 20:20:40.180384 2433 raft_consensus.cc:348] T 1716ac367ee64917bb929aebb8ee9126 P 46215a0dc4cc40e48eafa1ac007e1e06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19a96e445738489db15cbc247b2ad210" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 43375 } } peers { permanent_uuid: "46215a0dc4cc40e48eafa1ac007e1e06" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41347 } } peers { permanent_uuid: "53ac50f050ee4a3681e16f45546e4b72" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 43763 } }
I20260620 20:20:40.182178 2433 raft_consensus.cc:374] T 1716ac367ee64917bb929aebb8ee9126 P 46215a0dc4cc40e48eafa1ac007e1e06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:20:40.182296 2433 raft_consensus.cc:729] T 1716ac367ee64917bb929aebb8ee9126 P 46215a0dc4cc40e48eafa1ac007e1e06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46215a0dc4cc40e48eafa1ac007e1e06, State: Initialized, Role: FOLLOWER
I20260620 20:20:40.182940 2433 consensus_queue.cc:260] T 1716ac367ee64917bb929aebb8ee9126 P 46215a0dc4cc40e48eafa1ac007e1e06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19a96e445738489db15cbc247b2ad210" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 43375 } } peers { permanent_uuid: "46215a0dc4cc40e48eafa1ac007e1e06" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41347 } } peers { permanent_uuid: "53ac50f050ee4a3681e16f45546e4b72" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 43763 } }
I20260620 20:20:40.184135 2412 heartbeater.cc:503] Master 127.16.217.62:45647 was elected leader, sending a full tablet report...
I20260620 20:20:40.184654 2433 ts_tablet_manager.cc:1428] T 1716ac367ee64917bb929aebb8ee9126 P 46215a0dc4cc40e48eafa1ac007e1e06: Time spent starting tablet: real 0.017s user 0.012s sys 0.003s
W20260620 20:20:40.226327 2145 tablet.cc:2369] T 1716ac367ee64917bb929aebb8ee9126 P 19a96e445738489db15cbc247b2ad210: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 20:20:40.264166 2279 tablet.cc:2369] T 1716ac367ee64917bb929aebb8ee9126 P 53ac50f050ee4a3681e16f45546e4b72: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 20:20:40.278285 2438 raft_consensus.cc:482] T 1716ac367ee64917bb929aebb8ee9126 P 19a96e445738489db15cbc247b2ad210 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 20:20:40.278672 2438 raft_consensus.cc:504] T 1716ac367ee64917bb929aebb8ee9126 P 19a96e445738489db15cbc247b2ad210 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19a96e445738489db15cbc247b2ad210" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 43375 } } peers { permanent_uuid: "46215a0dc4cc40e48eafa1ac007e1e06" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41347 } } peers { permanent_uuid: "53ac50f050ee4a3681e16f45546e4b72" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 43763 } }
I20260620 20:20:40.281051 2438 leader_election.cc:288] T 1716ac367ee64917bb929aebb8ee9126 P 19a96e445738489db15cbc247b2ad210 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 46215a0dc4cc40e48eafa1ac007e1e06 (127.16.217.3:41347), 53ac50f050ee4a3681e16f45546e4b72 (127.16.217.2:43763)
W20260620 20:20:40.287222 2413 tablet.cc:2369] T 1716ac367ee64917bb929aebb8ee9126 P 46215a0dc4cc40e48eafa1ac007e1e06: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 20:20:40.290323 2428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:40.282132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.217.1:36157 (local address 127.16.217.3:41347)
0620 20:20:40.282363 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0620 20:20:40.282371 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 20:20:40.283538 (+ 1167us) server_negotiation.cc:407] Connection header received
0620 20:20:40.283828 (+ 290us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:20:40.283835 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:20:40.283922 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:20:40.284117 (+ 195us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:20:40.284723 (+ 606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:40.285456 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:40.288547 (+ 3091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:40.289308 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:40.289410 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:20:40.289461 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0620 20:20:40.289984 (+ 523us) server_negotiation.cc:299] Negotiation successful
0620 20:20:40.290083 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260620 20:20:40.290541 2440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:40.281392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.3:41347 (local address 127.16.217.1:36157)
0620 20:20:40.283398 (+ 2006us) negotiation.cc:107] Waiting for socket to connect
0620 20:20:40.283417 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 20:20:40.283639 (+ 222us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:20:40.284125 (+ 486us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:20:40.284129 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:20:40.284168 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:20:40.284549 (+ 381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:20:40.284557 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:20:40.285600 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:20:40.285608 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:20:40.288363 (+ 2755us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:20:40.288378 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:20:40.289611 (+ 1233us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:20:40.289616 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:20:40.289770 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:20:40.289813 (+ 43us) client_negotiation.cc:769] Sending connection context
0620 20:20:40.289950 (+ 137us) client_negotiation.cc:240] Negotiation successful
0620 20:20:40.290092 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1892,"mutex_wait_us":192,"thread_start_us":1306,"threads_started":1}
I20260620 20:20:40.290702 2441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:40.281334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.2:43763 (local address 127.16.217.1:53813)
0620 20:20:40.283074 (+ 1740us) negotiation.cc:107] Waiting for socket to connect
0620 20:20:40.283101 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0620 20:20:40.283369 (+ 268us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:20:40.283789 (+ 420us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:20:40.283795 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:20:40.283817 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:20:40.284233 (+ 416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:20:40.284244 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:20:40.285652 (+ 1408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:20:40.285661 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:20:40.288363 (+ 2702us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:20:40.288378 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:20:40.289902 (+ 1524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:20:40.289911 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:20:40.290077 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:20:40.290104 (+ 27us) client_negotiation.cc:769] Sending connection context
0620 20:20:40.290193 (+ 89us) client_negotiation.cc:240] Negotiation successful
0620 20:20:40.290319 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1574,"mutex_wait_us":136,"thread_start_us":1917,"threads_started":1}
I20260620 20:20:40.290860 2430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:40.281842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.217.1:53813 (local address 127.16.217.2:43763)
0620 20:20:40.282611 (+ 769us) server_negotiation.cc:206] Beginning negotiation
0620 20:20:40.282621 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 20:20:40.283221 (+ 600us) server_negotiation.cc:407] Connection header received
0620 20:20:40.283562 (+ 341us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:20:40.283570 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:20:40.283652 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:20:40.283740 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:20:40.284397 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:40.285532 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:40.288525 (+ 2993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:40.289658 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:40.289791 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:20:40.289839 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0620 20:20:40.290251 (+ 412us) server_negotiation.cc:299] Negotiation successful
0620 20:20:40.290590 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":594}
I20260620 20:20:40.291803 2367 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1716ac367ee64917bb929aebb8ee9126" candidate_uuid: "19a96e445738489db15cbc247b2ad210" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46215a0dc4cc40e48eafa1ac007e1e06" is_pre_election: true
I20260620 20:20:40.291819 2233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1716ac367ee64917bb929aebb8ee9126" candidate_uuid: "19a96e445738489db15cbc247b2ad210" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53ac50f050ee4a3681e16f45546e4b72" is_pre_election: true
I20260620 20:20:40.292279 2367 raft_consensus.cc:2454] T 1716ac367ee64917bb929aebb8ee9126 P 46215a0dc4cc40e48eafa1ac007e1e06 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 19a96e445738489db15cbc247b2ad210 in term 0.
I20260620 20:20:40.292280 2233 raft_consensus.cc:2454] T 1716ac367ee64917bb929aebb8ee9126 P 53ac50f050ee4a3681e16f45546e4b72 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 19a96e445738489db15cbc247b2ad210 in term 0.
I20260620 20:20:40.292928 2034 leader_election.cc:302] T 1716ac367ee64917bb929aebb8ee9126 P 19a96e445738489db15cbc247b2ad210 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 19a96e445738489db15cbc247b2ad210, 53ac50f050ee4a3681e16f45546e4b72; no voters:
I20260620 20:20:40.293354 2438 raft_consensus.cc:2781] T 1716ac367ee64917bb929aebb8ee9126 P 19a96e445738489db15cbc247b2ad210 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 20:20:40.293499 2438 raft_consensus.cc:482] T 1716ac367ee64917bb929aebb8ee9126 P 19a96e445738489db15cbc247b2ad210 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 20:20:40.293606 2438 raft_consensus.cc:3037] T 1716ac367ee64917bb929aebb8ee9126 P 19a96e445738489db15cbc247b2ad210 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:20:40.295895 2438 raft_consensus.cc:504] T 1716ac367ee64917bb929aebb8ee9126 P 19a96e445738489db15cbc247b2ad210 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19a96e445738489db15cbc247b2ad210" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 43375 } } peers { permanent_uuid: "46215a0dc4cc40e48eafa1ac007e1e06" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41347 } } peers { permanent_uuid: "53ac50f050ee4a3681e16f45546e4b72" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 43763 } }
I20260620 20:20:40.296598 2438 leader_election.cc:288] T 1716ac367ee64917bb929aebb8ee9126 P 19a96e445738489db15cbc247b2ad210 [CANDIDATE]: Term 1 election: Requested vote from peers 46215a0dc4cc40e48eafa1ac007e1e06 (127.16.217.3:41347), 53ac50f050ee4a3681e16f45546e4b72 (127.16.217.2:43763)
I20260620 20:20:40.297056 2367 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1716ac367ee64917bb929aebb8ee9126" candidate_uuid: "19a96e445738489db15cbc247b2ad210" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46215a0dc4cc40e48eafa1ac007e1e06"
I20260620 20:20:40.297123 2233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1716ac367ee64917bb929aebb8ee9126" candidate_uuid: "19a96e445738489db15cbc247b2ad210" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53ac50f050ee4a3681e16f45546e4b72"
I20260620 20:20:40.297283 2367 raft_consensus.cc:3037] T 1716ac367ee64917bb929aebb8ee9126 P 46215a0dc4cc40e48eafa1ac007e1e06 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:20:40.297312 2233 raft_consensus.cc:3037] T 1716ac367ee64917bb929aebb8ee9126 P 53ac50f050ee4a3681e16f45546e4b72 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:20:40.300544 2233 raft_consensus.cc:2454] T 1716ac367ee64917bb929aebb8ee9126 P 53ac50f050ee4a3681e16f45546e4b72 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 19a96e445738489db15cbc247b2ad210 in term 1.
I20260620 20:20:40.300544 2367 raft_consensus.cc:2454] T 1716ac367ee64917bb929aebb8ee9126 P 46215a0dc4cc40e48eafa1ac007e1e06 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 19a96e445738489db15cbc247b2ad210 in term 1.
I20260620 20:20:40.300987 2032 leader_election.cc:302] T 1716ac367ee64917bb929aebb8ee9126 P 19a96e445738489db15cbc247b2ad210 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 19a96e445738489db15cbc247b2ad210, 46215a0dc4cc40e48eafa1ac007e1e06, 53ac50f050ee4a3681e16f45546e4b72; no voters:
I20260620 20:20:40.301347 2438 raft_consensus.cc:2781] T 1716ac367ee64917bb929aebb8ee9126 P 19a96e445738489db15cbc247b2ad210 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:20:40.301842 2438 raft_consensus.cc:686] T 1716ac367ee64917bb929aebb8ee9126 P 19a96e445738489db15cbc247b2ad210 [term 1 LEADER]: Becoming Leader. State: Replica: 19a96e445738489db15cbc247b2ad210, State: Running, Role: LEADER
I20260620 20:20:40.302628 2438 consensus_queue.cc:237] T 1716ac367ee64917bb929aebb8ee9126 P 19a96e445738489db15cbc247b2ad210 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19a96e445738489db15cbc247b2ad210" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 43375 } } peers { permanent_uuid: "46215a0dc4cc40e48eafa1ac007e1e06" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41347 } } peers { permanent_uuid: "53ac50f050ee4a3681e16f45546e4b72" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 43763 } }
I20260620 20:20:40.306941 1955 catalog_manager.cc:5495] T 1716ac367ee64917bb929aebb8ee9126 P 19a96e445738489db15cbc247b2ad210 reported cstate change: term changed from 0 to 1, leader changed from <none> to 19a96e445738489db15cbc247b2ad210 (127.16.217.1). New cstate: current_term: 1 leader_uuid: "19a96e445738489db15cbc247b2ad210" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19a96e445738489db15cbc247b2ad210" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 43375 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "46215a0dc4cc40e48eafa1ac007e1e06" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41347 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "53ac50f050ee4a3681e16f45546e4b72" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 43763 } health_report { overall_health: UNKNOWN } } }
I20260620 20:20:40.357960 2426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:40.352850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38230 (local address 127.16.217.1:43375)
0620 20:20:40.353039 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0620 20:20:40.353043 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:20:40.353115 (+ 72us) server_negotiation.cc:407] Connection header received
0620 20:20:40.353290 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:20:40.353293 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:20:40.353392 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:20:40.353514 (+ 122us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 20:20:40.353881 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:40.354607 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:40.355963 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:40.356418 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:40.356509 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:20:40.356930 (+ 421us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 20:20:40.357454 (+ 524us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 20:20:40.357560 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0620 20:20:40.357636 (+ 76us) server_negotiation.cc:299] Negotiation successful
0620 20:20:40.357701 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 20:20:40.366207 2233 raft_consensus.cc:1261] T 1716ac367ee64917bb929aebb8ee9126 P 53ac50f050ee4a3681e16f45546e4b72 [term 1 FOLLOWER]: Refusing update from remote peer 19a96e445738489db15cbc247b2ad210: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:20:40.366214 2367 raft_consensus.cc:1261] T 1716ac367ee64917bb929aebb8ee9126 P 46215a0dc4cc40e48eafa1ac007e1e06 [term 1 FOLLOWER]: Refusing update from remote peer 19a96e445738489db15cbc247b2ad210: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:20:40.367236 2442 consensus_queue.cc:1035] T 1716ac367ee64917bb929aebb8ee9126 P 19a96e445738489db15cbc247b2ad210 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46215a0dc4cc40e48eafa1ac007e1e06" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41347 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:20:40.367575 2438 consensus_queue.cc:1035] T 1716ac367ee64917bb929aebb8ee9126 P 19a96e445738489db15cbc247b2ad210 [LEADER]: Connected to new peer: Peer: permanent_uuid: "53ac50f050ee4a3681e16f45546e4b72" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 43763 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:20:40.381243 2446 mvcc.cc:205] Tried to move back new op lower bound from 7299018098129420288 to 7299018097883201536. Current Snapshot: MvccSnapshot[applied={T|T < 7299018098129420288}]
I20260620 20:20:40.385210 2447 mvcc.cc:205] Tried to move back new op lower bound from 7299018098129420288 to 7299018097883201536. Current Snapshot: MvccSnapshot[applied={T|T < 7299018098129420288}]
I20260620 20:20:40.395767 1954 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52000:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 20:20:40.396113 1954 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52000:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 20:20:40.399973 1954 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 859fecb705a1417fa625c6317c298836: Sending DeleteTablet for 3 replicas of tablet 1716ac367ee64917bb929aebb8ee9126
I20260620 20:20:40.401113 2079 tablet_service.cc:1514] Processing DeleteTablet for tablet 1716ac367ee64917bb929aebb8ee9126 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:20:40 UTC) from {username='kudu', principal='kudu/127.16.217.62@KRBTEST.COM'} at 127.0.0.1:38228
I20260620 20:20:40.401118 2347 tablet_service.cc:1514] Processing DeleteTablet for tablet 1716ac367ee64917bb929aebb8ee9126 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:20:40 UTC) from {username='kudu', principal='kudu/127.16.217.62@KRBTEST.COM'} at 127.0.0.1:57222
I20260620 20:20:40.401119 2213 tablet_service.cc:1514] Processing DeleteTablet for tablet 1716ac367ee64917bb929aebb8ee9126 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:20:40 UTC) from {username='kudu', principal='kudu/127.16.217.62@KRBTEST.COM'} at 127.0.0.1:60480
I20260620 20:20:40.403714 17252 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu pid 2016
I20260620 20:20:40.414695 2454 tablet_replica.cc:330] T 1716ac367ee64917bb929aebb8ee9126 P 46215a0dc4cc40e48eafa1ac007e1e06: stopping tablet replica
I20260620 20:20:40.415467 2454 raft_consensus.cc:2229] T 1716ac367ee64917bb929aebb8ee9126 P 46215a0dc4cc40e48eafa1ac007e1e06 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 20:20:40.416200 2454 raft_consensus.cc:2258] T 1716ac367ee64917bb929aebb8ee9126 P 46215a0dc4cc40e48eafa1ac007e1e06 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:20:40.418991 2456 tablet_replica.cc:330] T 1716ac367ee64917bb929aebb8ee9126 P 19a96e445738489db15cbc247b2ad210: stopping tablet replica
I20260620 20:20:40.419486 2456 raft_consensus.cc:2229] T 1716ac367ee64917bb929aebb8ee9126 P 19a96e445738489db15cbc247b2ad210 [term 1 LEADER]: Raft consensus shutting down.
I20260620 20:20:40.419813 2454 ts_tablet_manager.cc:1905] T 1716ac367ee64917bb929aebb8ee9126 P 46215a0dc4cc40e48eafa1ac007e1e06: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:20:40.420291 2456 raft_consensus.cc:2258] T 1716ac367ee64917bb929aebb8ee9126 P 19a96e445738489db15cbc247b2ad210 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:20:40.422780 2456 ts_tablet_manager.cc:1905] T 1716ac367ee64917bb929aebb8ee9126 P 19a96e445738489db15cbc247b2ad210: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:20:40.427276 2454 ts_tablet_manager.cc:1918] T 1716ac367ee64917bb929aebb8ee9126 P 46215a0dc4cc40e48eafa1ac007e1e06: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 20:20:40.427491 2454 log.cc:1196] T 1716ac367ee64917bb929aebb8ee9126 P 46215a0dc4cc40e48eafa1ac007e1e06: Deleting WAL directory at /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986536687152-17252-0/minicluster-data/ts-2/wal/wals/1716ac367ee64917bb929aebb8ee9126
I20260620 20:20:40.428108 2454 ts_tablet_manager.cc:1939] T 1716ac367ee64917bb929aebb8ee9126 P 46215a0dc4cc40e48eafa1ac007e1e06: Deleting consensus metadata
I20260620 20:20:40.430131 1939 catalog_manager.cc:4841] TS 46215a0dc4cc40e48eafa1ac007e1e06 (127.16.217.3:41347): tablet 1716ac367ee64917bb929aebb8ee9126 (table test-table [id=4d9c4343641f44be8ab32001654affeb]) successfully deleted
I20260620 20:20:40.431362 2456 ts_tablet_manager.cc:1918] T 1716ac367ee64917bb929aebb8ee9126 P 19a96e445738489db15cbc247b2ad210: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 20:20:40.431546 2456 log.cc:1196] T 1716ac367ee64917bb929aebb8ee9126 P 19a96e445738489db15cbc247b2ad210: Deleting WAL directory at /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986536687152-17252-0/minicluster-data/ts-0/wal/wals/1716ac367ee64917bb929aebb8ee9126
I20260620 20:20:40.432147 2455 tablet_replica.cc:330] T 1716ac367ee64917bb929aebb8ee9126 P 53ac50f050ee4a3681e16f45546e4b72: stopping tablet replica
I20260620 20:20:40.432170 2456 ts_tablet_manager.cc:1939] T 1716ac367ee64917bb929aebb8ee9126 P 19a96e445738489db15cbc247b2ad210: Deleting consensus metadata
I20260620 20:20:40.432729 2455 raft_consensus.cc:2229] T 1716ac367ee64917bb929aebb8ee9126 P 53ac50f050ee4a3681e16f45546e4b72 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 20:20:40.433297 2455 raft_consensus.cc:2258] T 1716ac367ee64917bb929aebb8ee9126 P 53ac50f050ee4a3681e16f45546e4b72 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:20:40.433919 1942 catalog_manager.cc:4841] TS 19a96e445738489db15cbc247b2ad210 (127.16.217.1:43375): tablet 1716ac367ee64917bb929aebb8ee9126 (table test-table [id=4d9c4343641f44be8ab32001654affeb]) successfully deleted
I20260620 20:20:40.435619 2455 ts_tablet_manager.cc:1905] T 1716ac367ee64917bb929aebb8ee9126 P 53ac50f050ee4a3681e16f45546e4b72: Deleting tablet data with delete state TABLET_DATA_DELETED
Jun 20 20:20:40 dist-test-slave-2hpg krb5kdc[1910](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986837, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.217.1@KRBTEST.COM
I20260620 20:20:40.441838 2455 ts_tablet_manager.cc:1918] T 1716ac367ee64917bb929aebb8ee9126 P 53ac50f050ee4a3681e16f45546e4b72: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 20:20:40.442054 2455 log.cc:1196] T 1716ac367ee64917bb929aebb8ee9126 P 53ac50f050ee4a3681e16f45546e4b72: Deleting WAL directory at /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986536687152-17252-0/minicluster-data/ts-1/wal/wals/1716ac367ee64917bb929aebb8ee9126
I20260620 20:20:40.442616 2455 ts_tablet_manager.cc:1939] T 1716ac367ee64917bb929aebb8ee9126 P 53ac50f050ee4a3681e16f45546e4b72: Deleting consensus metadata
I20260620 20:20:40.444547 1941 catalog_manager.cc:4841] TS 53ac50f050ee4a3681e16f45546e4b72 (127.16.217.2:43763): tablet 1716ac367ee64917bb929aebb8ee9126 (table test-table [id=4d9c4343641f44be8ab32001654affeb]) successfully deleted
I20260620 20:20:40.448210 2426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:40.405229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44494 (local address 127.16.217.1:43375)
0620 20:20:40.405450 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0620 20:20:40.405459 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 20:20:40.429053 (+ 23594us) server_negotiation.cc:407] Connection header received
0620 20:20:40.429577 (+ 524us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:20:40.429583 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:20:40.429680 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:20:40.429749 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:20:40.431646 (+ 1897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:40.432877 (+ 1231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:40.434967 (+ 2090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:40.435699 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:40.435825 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:20:40.441123 (+ 5298us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:20:40.441139 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:20:40.441158 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:20:40.441219 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:20:40.445180 (+ 3961us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:20:40.446066 (+ 886us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:20:40.446072 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:20:40.446075 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:20:40.446228 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:20:40.446639 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:20:40.446642 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:20:40.446644 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:20:40.446988 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:20:40.447220 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0620 20:20:40.447628 (+ 408us) server_negotiation.cc:299] Negotiation successful
0620 20:20:40.447805 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 20:20:40.448939 2139 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:20:40.550371 17252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu with pid 2016
I20260620 20:20:40.568982 17252 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu pid 2150
Jun 20 20:20:40 dist-test-slave-2hpg krb5kdc[1910](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986837, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.217.2@KRBTEST.COM
I20260620 20:20:40.584830 2430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:40.570019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45130 (local address 127.16.217.2:43763)
0620 20:20:40.570213 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0620 20:20:40.570218 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:20:40.570237 (+ 19us) server_negotiation.cc:407] Connection header received
0620 20:20:40.570402 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:20:40.570406 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:20:40.570471 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:20:40.570597 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:20:40.571985 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:40.572851 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:40.574223 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:40.574665 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:40.574770 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:20:40.578964 (+ 4194us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:20:40.578979 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:20:40.578997 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:20:40.579039 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:20:40.581937 (+ 2898us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:20:40.582642 (+ 705us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:20:40.582650 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:20:40.582652 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:20:40.582772 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:20:40.583116 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:20:40.583119 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:20:40.583121 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:20:40.583630 (+ 509us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:20:40.583859 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0620 20:20:40.584202 (+ 343us) server_negotiation.cc:299] Negotiation successful
0620 20:20:40.584373 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260620 20:20:40.585625 2273 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:20:40.677690 17252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu with pid 2150
I20260620 20:20:40.695008 17252 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu pid 2284
Jun 20 20:20:40 dist-test-slave-2hpg krb5kdc[1910](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986837, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.217.3@KRBTEST.COM
I20260620 20:20:40.710188 2428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:40.695862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54602 (local address 127.16.217.3:41347)
0620 20:20:40.696052 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0620 20:20:40.696061 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 20:20:40.696094 (+ 33us) server_negotiation.cc:407] Connection header received
0620 20:20:40.696314 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:20:40.696318 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:20:40.696384 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:20:40.696486 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:20:40.697731 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:40.698520 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:40.699761 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:40.700152 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:40.700284 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:20:40.704514 (+ 4230us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:20:40.704529 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:20:40.704547 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:20:40.704584 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:20:40.707282 (+ 2698us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:20:40.707936 (+ 654us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:20:40.707944 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:20:40.707946 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:20:40.708095 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:20:40.708460 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:20:40.708466 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:20:40.708469 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:20:40.708801 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:20:40.709107 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0620 20:20:40.709403 (+ 296us) server_negotiation.cc:299] Negotiation successful
0620 20:20:40.709614 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 20:20:40.710875 2407 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:20:40.798754 17252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu with pid 2284
I20260620 20:20:40.815868 17252 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu pid 1923
I20260620 20:20:40.817027 1985 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:20:40.930269 17252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu with pid 1923
2026-06-20T20:20:40Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (4458 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-06-20T20:20:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T20:20:40Z Disabled control of system clock
I20260620 20:20:41.008563 17252 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
/tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986536687152-17252-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986536687152-17252-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986536687152-17252-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986536687152-17252-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.217.62:41191
--webserver_interface=127.16.217.62
--webserver_port=0
--builtin_ntp_servers=127.16.217.20:35449
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.217.62:41191
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 20:20:41.339332 2465 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:20:41.339632 2465 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:20:41.339787 2465 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:20:41.346976 2465 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:20:41.347097 2465 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:20:41.347137 2465 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:20:41.347187 2465 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:20:41.347234 2465 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:20:41.356413 2465 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.217.20:35449
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986536687152-17252-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986536687152-17252-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.217.62:41191
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.217.62:41191
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986536687152-17252-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.217.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986536687152-17252-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:20:41.357672 2465 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:20:41.359553 2465 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:20:41.371306 2470 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:20:41.371381 2473 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:20:41.371358 2471 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:20:41.372195 2465 server_base.cc:1034] running on GCE node
I20260620 20:20:41.373037 2465 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:20:41.374497 2465 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:20:41.375720 2465 hybrid_clock.cc:648] HybridClock initialized: now 1781986841375674 us; error 45 us; skew 500 ppm
I20260620 20:20:41.376063 2465 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 20:20:41.382951 2465 webserver.cc:460] Webserver started at http://127.16.217.62:45409/ using document root <none> and password file <none>
I20260620 20:20:41.383701 2465 fs_manager.cc:359] Metadata directory not provided
I20260620 20:20:41.383837 2465 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:20:41.384174 2465 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:20:41.387315 2465 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986536687152-17252-0/minicluster-data/master-0/data/instance:
uuid: "f576a95bc9ee4f509de5fbac6e6d9758"
format_stamp: "Formatted at 2026-06-20 20:20:41 on dist-test-slave-2hpg"
server_key: "718c27f8236665101761fd1f5c8aba18"
server_key_iv: "fab9637662e44d21e1c2ce95e1af859a"
server_key_version: "encryptionkey@0"
I20260620 20:20:41.388139 2465 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986536687152-17252-0/minicluster-data/master-0/wal/instance:
uuid: "f576a95bc9ee4f509de5fbac6e6d9758"
format_stamp: "Formatted at 2026-06-20 20:20:41 on dist-test-slave-2hpg"
server_key: "718c27f8236665101761fd1f5c8aba18"
server_key_iv: "fab9637662e44d21e1c2ce95e1af859a"
server_key_version: "encryptionkey@0"
I20260620 20:20:41.393340 2465 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 20:20:41.396808 2479 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:20:41.398193 2465 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 20:20:41.398370 2465 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986536687152-17252-0/minicluster-data/master-0/data,/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986536687152-17252-0/minicluster-data/master-0/wal
uuid: "f576a95bc9ee4f509de5fbac6e6d9758"
format_stamp: "Formatted at 2026-06-20 20:20:41 on dist-test-slave-2hpg"
server_key: "718c27f8236665101761fd1f5c8aba18"
server_key_iv: "fab9637662e44d21e1c2ce95e1af859a"
server_key_version: "encryptionkey@0"
I20260620 20:20:41.398582 2465 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986536687152-17252-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986536687152-17252-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986536687152-17252-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:20:41.416832 2465 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:20:41.417673 2465 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:20:41.418027 2465 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:20:41.437505 2465 rpc_server.cc:307] RPC server started. Bound to: 127.16.217.62:41191
I20260620 20:20:41.437517 2530 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.217.62:41191 every 8 connection(s)
I20260620 20:20:41.438898 2465 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986536687152-17252-0/minicluster-data/master-0/data/info.pb
I20260620 20:20:41.441720 17252 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu as pid 2465
I20260620 20:20:41.441931 17252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986536687152-17252-0/minicluster-data/master-0/wal/instance
I20260620 20:20:41.442396 17252 external_mini_cluster.cc:1428] Setting key 5ba60dd2094c4f3a3d4bd73576a09032
I20260620 20:20:41.442873 2531 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:20:41.455987 2531 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f576a95bc9ee4f509de5fbac6e6d9758: Bootstrap starting.
I20260620 20:20:41.456491 2534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:41.444294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44212 (local address 127.16.217.62:41191)
0620 20:20:41.445274 (+ 980us) server_negotiation.cc:206] Beginning negotiation
0620 20:20:41.445285 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 20:20:41.445327 (+ 42us) server_negotiation.cc:407] Connection header received
0620 20:20:41.447747 (+ 2420us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:20:41.447769 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:20:41.448236 (+ 467us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:20:41.448651 (+ 415us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:20:41.449223 (+ 572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:41.450585 (+ 1362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:41.451931 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:41.452663 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:41.452862 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:20:41.454160 (+ 1298us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:20:41.454168 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:20:41.454181 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:20:41.454733 (+ 552us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:20:41.454815 (+ 82us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:20:41.454836 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:20:41.454965 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0620 20:20:41.455181 (+ 216us) server_negotiation.cc:299] Negotiation successful
0620 20:20:41.455486 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":585,"thread_start_us":365,"threads_started":1}
I20260620 20:20:41.459367 2531 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f576a95bc9ee4f509de5fbac6e6d9758: Neither blocks nor log segments found. Creating new log.
I20260620 20:20:41.460400 2531 log.cc:824] T 00000000000000000000000000000000 P f576a95bc9ee4f509de5fbac6e6d9758: Log is configured to *not* fsync() on all Append() calls
I20260620 20:20:41.463438 2531 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f576a95bc9ee4f509de5fbac6e6d9758: No bootstrap required, opened a new log
I20260620 20:20:41.471191 2531 raft_consensus.cc:348] T 00000000000000000000000000000000 P f576a95bc9ee4f509de5fbac6e6d9758 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f576a95bc9ee4f509de5fbac6e6d9758" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 41191 } }
I20260620 20:20:41.471549 2531 raft_consensus.cc:374] T 00000000000000000000000000000000 P f576a95bc9ee4f509de5fbac6e6d9758 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:20:41.471629 2531 raft_consensus.cc:729] T 00000000000000000000000000000000 P f576a95bc9ee4f509de5fbac6e6d9758 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f576a95bc9ee4f509de5fbac6e6d9758, State: Initialized, Role: FOLLOWER
I20260620 20:20:41.472139 2531 consensus_queue.cc:260] T 00000000000000000000000000000000 P f576a95bc9ee4f509de5fbac6e6d9758 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f576a95bc9ee4f509de5fbac6e6d9758" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 41191 } }
I20260620 20:20:41.472330 2531 raft_consensus.cc:388] T 00000000000000000000000000000000 P f576a95bc9ee4f509de5fbac6e6d9758 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 20:20:41.472427 2531 raft_consensus.cc:482] T 00000000000000000000000000000000 P f576a95bc9ee4f509de5fbac6e6d9758 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 20:20:41.472551 2531 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f576a95bc9ee4f509de5fbac6e6d9758 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:20:41.475133 2531 raft_consensus.cc:504] T 00000000000000000000000000000000 P f576a95bc9ee4f509de5fbac6e6d9758 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f576a95bc9ee4f509de5fbac6e6d9758" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 41191 } }
I20260620 20:20:41.475708 2531 leader_election.cc:302] T 00000000000000000000000000000000 P f576a95bc9ee4f509de5fbac6e6d9758 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f576a95bc9ee4f509de5fbac6e6d9758; no voters:
I20260620 20:20:41.476239 2531 leader_election.cc:288] T 00000000000000000000000000000000 P f576a95bc9ee4f509de5fbac6e6d9758 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 20:20:41.476439 2536 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f576a95bc9ee4f509de5fbac6e6d9758 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:20:41.477190 2536 raft_consensus.cc:686] T 00000000000000000000000000000000 P f576a95bc9ee4f509de5fbac6e6d9758 [term 1 LEADER]: Becoming Leader. State: Replica: f576a95bc9ee4f509de5fbac6e6d9758, State: Running, Role: LEADER
I20260620 20:20:41.477730 2536 consensus_queue.cc:237] T 00000000000000000000000000000000 P f576a95bc9ee4f509de5fbac6e6d9758 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f576a95bc9ee4f509de5fbac6e6d9758" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 41191 } }
I20260620 20:20:41.478452 2531 sys_catalog.cc:564] T 00000000000000000000000000000000 P f576a95bc9ee4f509de5fbac6e6d9758 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:20:41.481132 2538 sys_catalog.cc:455] T 00000000000000000000000000000000 P f576a95bc9ee4f509de5fbac6e6d9758 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f576a95bc9ee4f509de5fbac6e6d9758. Latest consensus state: current_term: 1 leader_uuid: "f576a95bc9ee4f509de5fbac6e6d9758" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f576a95bc9ee4f509de5fbac6e6d9758" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 41191 } } }
I20260620 20:20:41.481683 2538 sys_catalog.cc:458] T 00000000000000000000000000000000 P f576a95bc9ee4f509de5fbac6e6d9758 [sys.catalog]: This master's current role is: LEADER
I20260620 20:20:41.481154 2537 sys_catalog.cc:455] T 00000000000000000000000000000000 P f576a95bc9ee4f509de5fbac6e6d9758 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f576a95bc9ee4f509de5fbac6e6d9758" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f576a95bc9ee4f509de5fbac6e6d9758" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 41191 } } }
I20260620 20:20:41.484951 2537 sys_catalog.cc:458] T 00000000000000000000000000000000 P f576a95bc9ee4f509de5fbac6e6d9758 [sys.catalog]: This master's current role is: LEADER
I20260620 20:20:41.485443 2545 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 20:20:41.489835 2545 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 20:20:41.500221 2545 catalog_manager.cc:1347] Generated new cluster ID: 75f2a4d52ce34de786e0db8811db2396
I20260620 20:20:41.500367 2545 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 20:20:41.523355 2545 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 20:20:41.524722 2545 catalog_manager.cc:1504] Loading token signing keys...
I20260620 20:20:41.533701 2545 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f576a95bc9ee4f509de5fbac6e6d9758: Generated new TSK 0
I20260620 20:20:41.534596 2545 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 20:20:41.549203 17252 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
/tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986536687152-17252-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986536687152-17252-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986536687152-17252-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986536687152-17252-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.217.1:0
--local_ip_for_outbound_sockets=127.16.217.1
--webserver_interface=127.16.217.1
--webserver_port=0
--tserver_master_addrs=127.16.217.62:41191
--builtin_ntp_servers=127.16.217.20:35449
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 20:20:41.890746 2555 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:20:41.891034 2555 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:20:41.891188 2555 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:20:41.898485 2555 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:20:41.898590 2555 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:20:41.898751 2555 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.217.1
I20260620 20:20:41.908726 2555 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.217.20:35449
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986536687152-17252-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986536687152-17252-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.217.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986536687152-17252-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.217.1
--webserver_port=0
--tserver_master_addrs=127.16.217.62:41191
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.217.1
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986536687152-17252-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:20:41.910018 2555 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:20:41.911804 2555 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:20:41.925120 2561 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:20:41.925243 2560 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:20:41.925221 2563 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:20:41.925933 2555 server_base.cc:1034] running on GCE node
I20260620 20:20:41.926574 2555 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:20:41.927577 2555 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:20:41.928823 2555 hybrid_clock.cc:648] HybridClock initialized: now 1781986841928784 us; error 66 us; skew 500 ppm
I20260620 20:20:41.929163 2555 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 20:20:41.931532 2555 webserver.cc:460] Webserver started at http://127.16.217.1:37009/ using document root <none> and password file <none>
I20260620 20:20:41.932296 2555 fs_manager.cc:359] Metadata directory not provided
I20260620 20:20:41.932449 2555 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:20:41.932775 2555 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:20:41.935714 2555 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986536687152-17252-0/minicluster-data/ts-0/data/instance:
uuid: "cc6b356e5ea445c589807621489a8437"
format_stamp: "Formatted at 2026-06-20 20:20:41 on dist-test-slave-2hpg"
server_key: "2b20d8affb4122e233fe35ca2571e80c"
server_key_iv: "aba3a47850ccb428fc507b10144605b2"
server_key_version: "encryptionkey@0"
I20260620 20:20:41.936522 2555 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986536687152-17252-0/minicluster-data/ts-0/wal/instance:
uuid: "cc6b356e5ea445c589807621489a8437"
format_stamp: "Formatted at 2026-06-20 20:20:41 on dist-test-slave-2hpg"
server_key: "2b20d8affb4122e233fe35ca2571e80c"
server_key_iv: "aba3a47850ccb428fc507b10144605b2"
server_key_version: "encryptionkey@0"
I20260620 20:20:41.941769 2555 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260620 20:20:41.945482 2569 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:20:41.946949 2555 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 20:20:41.947144 2555 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986536687152-17252-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986536687152-17252-0/minicluster-data/ts-0/wal
uuid: "cc6b356e5ea445c589807621489a8437"
format_stamp: "Formatted at 2026-06-20 20:20:41 on dist-test-slave-2hpg"
server_key: "2b20d8affb4122e233fe35ca2571e80c"
server_key_iv: "aba3a47850ccb428fc507b10144605b2"
server_key_version: "encryptionkey@0"
I20260620 20:20:41.947372 2555 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986536687152-17252-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986536687152-17252-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986536687152-17252-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:20:41.975558 2555 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:20:41.976485 2555 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:20:41.976810 2555 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:20:41.977782 2555 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:20:41.979370 2555 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:20:41.979457 2555 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:20:41.979588 2555 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:20:41.979673 2555 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:20:42.013988 2555 rpc_server.cc:307] RPC server started. Bound to: 127.16.217.1:37071
I20260620 20:20:42.014048 2681 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.217.1:37071 every 8 connection(s)
I20260620 20:20:42.015367 2555 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986536687152-17252-0/minicluster-data/ts-0/data/info.pb
I20260620 20:20:42.023855 17252 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu as pid 2555
I20260620 20:20:42.024044 17252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986536687152-17252-0/minicluster-data/ts-0/wal/instance
I20260620 20:20:42.024505 17252 external_mini_cluster.cc:1428] Setting key 010af285d16b08c819d41fe00f5bc226
I20260620 20:20:42.027552 17252 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
/tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986536687152-17252-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986536687152-17252-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986536687152-17252-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986536687152-17252-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.217.2:0
--local_ip_for_outbound_sockets=127.16.217.2
--webserver_interface=127.16.217.2
--webserver_port=0
--tserver_master_addrs=127.16.217.62:41191
--builtin_ntp_servers=127.16.217.20:35449
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 20:20:42.028728 2684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:42.018233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.217.1:53983 (local address 127.16.217.62:41191)
0620 20:20:42.018695 (+ 462us) server_negotiation.cc:206] Beginning negotiation
0620 20:20:42.018700 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:20:42.020012 (+ 1312us) server_negotiation.cc:407] Connection header received
0620 20:20:42.021763 (+ 1751us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:20:42.021768 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:20:42.021839 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:20:42.022007 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:20:42.023129 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:42.023886 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:42.025754 (+ 1868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:42.026319 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:42.026455 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:20:42.027986 (+ 1531us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:20:42.027991 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:20:42.027994 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:20:42.028012 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:20:42.028051 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:20:42.028064 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:20:42.028134 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 20:20:42.028399 (+ 265us) server_negotiation.cc:299] Negotiation successful
0620 20:20:42.028473 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":354,"thread_start_us":239,"threads_started":1}
I20260620 20:20:42.030243 2685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:42.018906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:41191 (local address 127.16.217.1:53983)
0620 20:20:42.019852 (+ 946us) negotiation.cc:107] Waiting for socket to connect
0620 20:20:42.019888 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 20:20:42.021500 (+ 1612us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:20:42.022212 (+ 712us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:20:42.022220 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:20:42.022305 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:20:42.022940 (+ 635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:20:42.022967 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:20:42.024089 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:20:42.024094 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:20:42.025589 (+ 1495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:20:42.025601 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:20:42.026560 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:20:42.026568 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:20:42.026862 (+ 294us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:20:42.027625 (+ 763us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 20:20:42.027630 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:20:42.027696 (+ 66us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 20:20:42.027803 (+ 107us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:20:42.028167 (+ 364us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:20:42.028181 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:20:42.028192 (+ 11us) client_negotiation.cc:769] Sending connection context
0620 20:20:42.028387 (+ 195us) client_negotiation.cc:240] Negotiation successful
0620 20:20:42.028819 (+ 432us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":278,"threads_started":1}
I20260620 20:20:42.031872 2682 heartbeater.cc:347] Connected to a master server at 127.16.217.62:41191
I20260620 20:20:42.032172 2682 heartbeater.cc:464] Registering TS with master...
I20260620 20:20:42.032932 2682 heartbeater.cc:511] Master 127.16.217.62:41191 requested a full tablet report, sending...
I20260620 20:20:42.035203 2496 ts_manager.cc:194] Registered new tserver with Master: cc6b356e5ea445c589807621489a8437 (127.16.217.1:37071)
I20260620 20:20:42.036785 2496 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.217.1:53983
W20260620 20:20:42.355324 2686 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:20:42.355618 2686 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:20:42.355780 2686 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:20:42.363168 2686 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:20:42.363278 2686 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:20:42.363504 2686 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.217.2
I20260620 20:20:42.372972 2686 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.217.20:35449
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986536687152-17252-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986536687152-17252-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.217.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986536687152-17252-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.217.2
--webserver_port=0
--tserver_master_addrs=127.16.217.62:41191
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.217.2
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986536687152-17252-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:20:42.374389 2686 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:20:42.376204 2686 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:20:42.390332 2691 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:20:42.390300 2692 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:20:42.391264 2694 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:20:42.391714 2686 server_base.cc:1034] running on GCE node
I20260620 20:20:42.392300 2686 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:20:42.393247 2686 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:20:42.394443 2686 hybrid_clock.cc:648] HybridClock initialized: now 1781986842394401 us; error 38 us; skew 500 ppm
I20260620 20:20:42.394788 2686 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 20:20:42.397014 2686 webserver.cc:460] Webserver started at http://127.16.217.2:33821/ using document root <none> and password file <none>
I20260620 20:20:42.397680 2686 fs_manager.cc:359] Metadata directory not provided
I20260620 20:20:42.397791 2686 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:20:42.398131 2686 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:20:42.400837 2686 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986536687152-17252-0/minicluster-data/ts-1/data/instance:
uuid: "11b2590223ff4606ba04146e6d544c3e"
format_stamp: "Formatted at 2026-06-20 20:20:42 on dist-test-slave-2hpg"
server_key: "3b7ae15f589b62426f5ef9a15cd16c25"
server_key_iv: "ce11d51688cf931d7b82d5a7cfd49982"
server_key_version: "encryptionkey@0"
I20260620 20:20:42.401696 2686 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986536687152-17252-0/minicluster-data/ts-1/wal/instance:
uuid: "11b2590223ff4606ba04146e6d544c3e"
format_stamp: "Formatted at 2026-06-20 20:20:42 on dist-test-slave-2hpg"
server_key: "3b7ae15f589b62426f5ef9a15cd16c25"
server_key_iv: "ce11d51688cf931d7b82d5a7cfd49982"
server_key_version: "encryptionkey@0"
I20260620 20:20:42.406811 2686 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260620 20:20:42.410181 2700 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:20:42.411504 2686 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 20:20:42.411732 2686 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986536687152-17252-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986536687152-17252-0/minicluster-data/ts-1/wal
uuid: "11b2590223ff4606ba04146e6d544c3e"
format_stamp: "Formatted at 2026-06-20 20:20:42 on dist-test-slave-2hpg"
server_key: "3b7ae15f589b62426f5ef9a15cd16c25"
server_key_iv: "ce11d51688cf931d7b82d5a7cfd49982"
server_key_version: "encryptionkey@0"
I20260620 20:20:42.411949 2686 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986536687152-17252-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986536687152-17252-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986536687152-17252-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:20:42.445976 2686 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:20:42.446835 2686 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:20:42.447158 2686 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:20:42.448182 2686 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:20:42.449644 2686 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:20:42.449775 2686 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:20:42.449914 2686 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:20:42.450002 2686 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260620 20:20:42.485194 2686 rpc_server.cc:307] RPC server started. Bound to: 127.16.217.2:45795
I20260620 20:20:42.485286 2812 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.217.2:45795 every 8 connection(s)
I20260620 20:20:42.486696 2686 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986536687152-17252-0/minicluster-data/ts-1/data/info.pb
I20260620 20:20:42.494510 17252 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu as pid 2686
I20260620 20:20:42.494704 17252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986536687152-17252-0/minicluster-data/ts-1/wal/instance
I20260620 20:20:42.495306 17252 external_mini_cluster.cc:1428] Setting key 1150cb7572b148684574d38b76fb460f
I20260620 20:20:42.498803 17252 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
/tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986536687152-17252-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986536687152-17252-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986536687152-17252-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986536687152-17252-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.217.3:0
--local_ip_for_outbound_sockets=127.16.217.3
--webserver_interface=127.16.217.3
--webserver_port=0
--tserver_master_addrs=127.16.217.62:41191
--builtin_ntp_servers=127.16.217.20:35449
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 20:20:42.501022 2684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:42.489331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.217.2:40793 (local address 127.16.217.62:41191)
0620 20:20:42.489512 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0620 20:20:42.489516 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:20:42.491207 (+ 1691us) server_negotiation.cc:407] Connection header received
0620 20:20:42.492977 (+ 1770us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:20:42.492982 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:20:42.493062 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:20:42.493161 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:20:42.494287 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:42.495445 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:42.497599 (+ 2154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:42.498276 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:42.498425 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:20:42.500091 (+ 1666us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:20:42.500101 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:20:42.500109 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:20:42.500136 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:20:42.500180 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:20:42.500200 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:20:42.500295 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0620 20:20:42.500508 (+ 213us) server_negotiation.cc:299] Negotiation successful
0620 20:20:42.500614 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 20:20:42.502107 2815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:42.490014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:41191 (local address 127.16.217.2:40793)
0620 20:20:42.491057 (+ 1043us) negotiation.cc:107] Waiting for socket to connect
0620 20:20:42.491094 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 20:20:42.492712 (+ 1618us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:20:42.493363 (+ 651us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:20:42.493371 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:20:42.493454 (+ 83us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:20:42.494116 (+ 662us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:20:42.494133 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:20:42.495606 (+ 1473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:20:42.495610 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:20:42.497380 (+ 1770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:20:42.497403 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:20:42.498483 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:20:42.498488 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:20:42.498719 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:20:42.499743 (+ 1024us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 20:20:42.499746 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:20:42.499788 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 20:20:42.499863 (+ 75us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:20:42.500304 (+ 441us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:20:42.500312 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:20:42.500318 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 20:20:42.500467 (+ 149us) client_negotiation.cc:240] Negotiation successful
0620 20:20:42.500768 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":581,"thread_start_us":286,"threads_started":1}
I20260620 20:20:42.503582 2813 heartbeater.cc:347] Connected to a master server at 127.16.217.62:41191
I20260620 20:20:42.503886 2813 heartbeater.cc:464] Registering TS with master...
I20260620 20:20:42.504637 2813 heartbeater.cc:511] Master 127.16.217.62:41191 requested a full tablet report, sending...
I20260620 20:20:42.506166 2496 ts_manager.cc:194] Registered new tserver with Master: 11b2590223ff4606ba04146e6d544c3e (127.16.217.2:45795)
I20260620 20:20:42.506938 2496 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.217.2:40793
W20260620 20:20:42.834961 2816 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:20:42.835285 2816 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:20:42.835454 2816 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:20:42.842689 2816 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:20:42.842826 2816 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:20:42.843010 2816 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.217.3
I20260620 20:20:42.852270 2816 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.217.20:35449
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986536687152-17252-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986536687152-17252-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.217.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986536687152-17252-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.217.3
--webserver_port=0
--tserver_master_addrs=127.16.217.62:41191
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.217.3
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986536687152-17252-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:20:42.853581 2816 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:20:42.855369 2816 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:20:42.868978 2821 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:20:42.868968 2822 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:20:42.868973 2824 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:20:42.869782 2816 server_base.cc:1034] running on GCE node
I20260620 20:20:42.870442 2816 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:20:42.871373 2816 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:20:42.872597 2816 hybrid_clock.cc:648] HybridClock initialized: now 1781986842872537 us; error 46 us; skew 500 ppm
I20260620 20:20:42.872941 2816 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 20:20:42.875276 2816 webserver.cc:460] Webserver started at http://127.16.217.3:44169/ using document root <none> and password file <none>
I20260620 20:20:42.875996 2816 fs_manager.cc:359] Metadata directory not provided
I20260620 20:20:42.876083 2816 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:20:42.876441 2816 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:20:42.879565 2816 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986536687152-17252-0/minicluster-data/ts-2/data/instance:
uuid: "bcd959a07829492db7ded12f1952a44c"
format_stamp: "Formatted at 2026-06-20 20:20:42 on dist-test-slave-2hpg"
server_key: "3a319346097d5a79408c58ea2766e2ed"
server_key_iv: "8a50238fbf323971e46398a475908662"
server_key_version: "encryptionkey@0"
I20260620 20:20:42.880400 2816 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986536687152-17252-0/minicluster-data/ts-2/wal/instance:
uuid: "bcd959a07829492db7ded12f1952a44c"
format_stamp: "Formatted at 2026-06-20 20:20:42 on dist-test-slave-2hpg"
server_key: "3a319346097d5a79408c58ea2766e2ed"
server_key_iv: "8a50238fbf323971e46398a475908662"
server_key_version: "encryptionkey@0"
I20260620 20:20:42.886478 2816 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260620 20:20:42.890060 2830 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:20:42.891392 2816 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 20:20:42.891523 2816 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986536687152-17252-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986536687152-17252-0/minicluster-data/ts-2/wal
uuid: "bcd959a07829492db7ded12f1952a44c"
format_stamp: "Formatted at 2026-06-20 20:20:42 on dist-test-slave-2hpg"
server_key: "3a319346097d5a79408c58ea2766e2ed"
server_key_iv: "8a50238fbf323971e46398a475908662"
server_key_version: "encryptionkey@0"
I20260620 20:20:42.891700 2816 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986536687152-17252-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986536687152-17252-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986536687152-17252-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:20:42.928665 2816 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:20:42.929466 2816 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:20:42.929723 2816 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:20:42.930886 2816 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:20:42.932287 2816 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:20:42.932370 2816 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:20:42.932439 2816 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:20:42.932495 2816 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:20:42.965408 2816 rpc_server.cc:307] RPC server started. Bound to: 127.16.217.3:44433
I20260620 20:20:42.965454 2942 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.217.3:44433 every 8 connection(s)
I20260620 20:20:42.966805 2816 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986536687152-17252-0/minicluster-data/ts-2/data/info.pb
I20260620 20:20:42.974766 17252 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu as pid 2816
I20260620 20:20:42.974978 17252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986536687152-17252-0/minicluster-data/ts-2/wal/instance
I20260620 20:20:42.975457 17252 external_mini_cluster.cc:1428] Setting key 101bb96c235770536aa672c00d4cc8c7
I20260620 20:20:42.980162 2684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:42.969398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.217.3:50839 (local address 127.16.217.62:41191)
0620 20:20:42.969583 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0620 20:20:42.969588 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:20:42.971148 (+ 1560us) server_negotiation.cc:407] Connection header received
0620 20:20:42.972962 (+ 1814us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:20:42.972969 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:20:42.973077 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:20:42.973193 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:20:42.974374 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:42.975246 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:42.977209 (+ 1963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:42.977702 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:42.977805 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:20:42.979381 (+ 1576us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:20:42.979386 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:20:42.979389 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:20:42.979407 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:20:42.979440 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:20:42.979452 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:20:42.979523 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 20:20:42.979768 (+ 245us) server_negotiation.cc:299] Negotiation successful
0620 20:20:42.979901 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 20:20:42.981178 2945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:42.970049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:41191 (local address 127.16.217.3:50839)
0620 20:20:42.970988 (+ 939us) negotiation.cc:107] Waiting for socket to connect
0620 20:20:42.971024 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 20:20:42.972683 (+ 1659us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:20:42.973429 (+ 746us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:20:42.973438 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:20:42.973531 (+ 93us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:20:42.974205 (+ 674us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:20:42.974223 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:20:42.975400 (+ 1177us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:20:42.975405 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:20:42.977040 (+ 1635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:20:42.977056 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:20:42.977952 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:20:42.977960 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:20:42.978280 (+ 320us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:20:42.979074 (+ 794us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 20:20:42.979077 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:20:42.979117 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 20:20:42.979220 (+ 103us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:20:42.979538 (+ 318us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:20:42.979546 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:20:42.979560 (+ 14us) client_negotiation.cc:769] Sending connection context
0620 20:20:42.979705 (+ 145us) client_negotiation.cc:240] Negotiation successful
0620 20:20:42.980147 (+ 442us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":475,"thread_start_us":265,"threads_started":1}
I20260620 20:20:42.982558 2943 heartbeater.cc:347] Connected to a master server at 127.16.217.62:41191
I20260620 20:20:42.982878 2943 heartbeater.cc:464] Registering TS with master...
I20260620 20:20:42.983559 2943 heartbeater.cc:511] Master 127.16.217.62:41191 requested a full tablet report, sending...
I20260620 20:20:42.984921 2496 ts_manager.cc:194] Registered new tserver with Master: bcd959a07829492db7ded12f1952a44c (127.16.217.3:44433)
I20260620 20:20:42.985746 2496 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.217.3:50839
I20260620 20:20:42.990715 17252 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 20:20:43.001441 2684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:42.994916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44224 (local address 127.16.217.62:41191)
0620 20:20:42.995328 (+ 412us) server_negotiation.cc:206] Beginning negotiation
0620 20:20:42.995336 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 20:20:42.995668 (+ 332us) server_negotiation.cc:407] Connection header received
0620 20:20:42.995888 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:20:42.995895 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:20:42.996018 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:20:42.996127 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:20:42.996673 (+ 546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:42.997421 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:42.998959 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:42.999426 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:42.999585 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:20:43.000701 (+ 1116us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:20:43.000706 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:20:43.000710 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:20:43.000726 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:20:43.000757 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:20:43.000770 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:20:43.000870 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0620 20:20:43.001027 (+ 157us) server_negotiation.cc:299] Negotiation successful
0620 20:20:43.001123 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260620 20:20:43.005395 2496 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44224:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 20:20:43.012450 2496 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 20:20:43.040262 2682 heartbeater.cc:503] Master 127.16.217.62:41191 was elected leader, sending a full tablet report...
I20260620 20:20:43.049332 2955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:43.025743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37560 (local address 127.16.217.1:37071)
0620 20:20:43.039847 (+ 14104us) server_negotiation.cc:206] Beginning negotiation
0620 20:20:43.039857 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 20:20:43.039888 (+ 31us) server_negotiation.cc:407] Connection header received
0620 20:20:43.039995 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:20:43.040001 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:20:43.040259 (+ 258us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:20:43.040533 (+ 274us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:20:43.041561 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:43.043274 (+ 1713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:43.046481 (+ 3207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:43.047867 (+ 1386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:43.048006 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:20:43.048287 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0620 20:20:43.048770 (+ 483us) server_negotiation.cc:299] Negotiation successful
0620 20:20:43.048937 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13929,"thread_start_us":13816,"threads_started":1}
I20260620 20:20:43.049401 2958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:43.025681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.1:37071 (local address 127.0.0.1:37560)
0620 20:20:43.037207 (+ 11526us) negotiation.cc:107] Waiting for socket to connect
0620 20:20:43.037263 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0620 20:20:43.037450 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:20:43.040573 (+ 3123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:20:43.040578 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:20:43.040601 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:20:43.040961 (+ 360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:20:43.040975 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:20:43.043409 (+ 2434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:20:43.043414 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:20:43.046265 (+ 2851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:20:43.046282 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:20:43.048133 (+ 1851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:20:43.048139 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:20:43.048418 (+ 279us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:20:43.048566 (+ 148us) client_negotiation.cc:769] Sending connection context
0620 20:20:43.048699 (+ 133us) client_negotiation.cc:240] Negotiation successful
0620 20:20:43.048952 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11212,"mutex_wait_us":236,"thread_start_us":12070,"threads_started":1}
I20260620 20:20:43.050000 2956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:43.025754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.3:44433 (local address 127.0.0.1:43788)
0620 20:20:43.038045 (+ 12291us) negotiation.cc:107] Waiting for socket to connect
0620 20:20:43.038063 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 20:20:43.038192 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:20:43.040902 (+ 2710us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:20:43.040911 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:20:43.040939 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:20:43.042069 (+ 1130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:20:43.042083 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:20:43.043947 (+ 1864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:20:43.043952 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:20:43.046167 (+ 2215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:20:43.046181 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:20:43.048092 (+ 1911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:20:43.048101 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:20:43.048414 (+ 313us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:20:43.048566 (+ 152us) client_negotiation.cc:769] Sending connection context
0620 20:20:43.048699 (+ 133us) client_negotiation.cc:240] Negotiation successful
0620 20:20:43.048952 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12155,"mutex_wait_us":212,"thread_start_us":12726,"threads_started":1}
I20260620 20:20:43.051049 2957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:43.026214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.2:45795 (local address 127.0.0.1:53148)
0620 20:20:43.038761 (+ 12547us) negotiation.cc:107] Waiting for socket to connect
0620 20:20:43.038782 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 20:20:43.038903 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:20:43.040172 (+ 1269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:20:43.040180 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:20:43.040216 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:20:43.041436 (+ 1220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:20:43.041449 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:20:43.043246 (+ 1797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:20:43.043255 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:20:43.046294 (+ 3039us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:20:43.046305 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:20:43.050263 (+ 3958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:20:43.050269 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:20:43.050462 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:20:43.050528 (+ 66us) client_negotiation.cc:769] Sending connection context
0620 20:20:43.050618 (+ 90us) client_negotiation.cc:240] Negotiation successful
0620 20:20:43.050738 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12399,"mutex_wait_us":183,"thread_start_us":11475,"threads_started":1}
I20260620 20:20:43.051708 2959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:43.026577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43788 (local address 127.16.217.3:44433)
0620 20:20:43.040194 (+ 13617us) server_negotiation.cc:206] Beginning negotiation
0620 20:20:43.040209 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 20:20:43.040239 (+ 30us) server_negotiation.cc:407] Connection header received
0620 20:20:43.040328 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:20:43.040335 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:20:43.040540 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:20:43.040759 (+ 219us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:20:43.042507 (+ 1748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:43.043810 (+ 1303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:43.046362 (+ 2552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:43.047867 (+ 1505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:43.048005 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:20:43.048287 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0620 20:20:43.049970 (+ 1683us) server_negotiation.cc:299] Negotiation successful
0620 20:20:43.051309 (+ 1339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13448,"thread_start_us":13298,"threads_started":1}
I20260620 20:20:43.051779 2960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:43.027167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53148 (local address 127.16.217.2:45795)
0620 20:20:43.039419 (+ 12252us) server_negotiation.cc:206] Beginning negotiation
0620 20:20:43.039431 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 20:20:43.039472 (+ 41us) server_negotiation.cc:407] Connection header received
0620 20:20:43.039606 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:20:43.039619 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:20:43.039929 (+ 310us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:20:43.041636 (+ 1707us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:20:43.041893 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:43.043117 (+ 1224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:43.048375 (+ 5258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:43.049675 (+ 1300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:43.049814 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:20:43.050111 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0620 20:20:43.051280 (+ 1169us) server_negotiation.cc:299] Negotiation successful
0620 20:20:43.051446 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12063,"thread_start_us":11947,"threads_started":1}
I20260620 20:20:43.054157 2617 tablet_service.cc:1467] Processing CreateTablet for tablet 66e8a939d3cd4596832c379776c27092 (DEFAULT_TABLE table=test-table [id=325b285d855f4ac198d2b6bbabc601f9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:20:43.055019 2748 tablet_service.cc:1467] Processing CreateTablet for tablet 66e8a939d3cd4596832c379776c27092 (DEFAULT_TABLE table=test-table [id=325b285d855f4ac198d2b6bbabc601f9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:20:43.055648 2617 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 66e8a939d3cd4596832c379776c27092. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:20:43.056142 2748 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 66e8a939d3cd4596832c379776c27092. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:20:43.055907 2878 tablet_service.cc:1467] Processing CreateTablet for tablet 66e8a939d3cd4596832c379776c27092 (DEFAULT_TABLE table=test-table [id=325b285d855f4ac198d2b6bbabc601f9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:20:43.057238 2878 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 66e8a939d3cd4596832c379776c27092. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:20:43.068544 2961 tablet_bootstrap.cc:492] T 66e8a939d3cd4596832c379776c27092 P 11b2590223ff4606ba04146e6d544c3e: Bootstrap starting.
I20260620 20:20:43.070726 2962 tablet_bootstrap.cc:492] T 66e8a939d3cd4596832c379776c27092 P cc6b356e5ea445c589807621489a8437: Bootstrap starting.
I20260620 20:20:43.072190 2963 tablet_bootstrap.cc:492] T 66e8a939d3cd4596832c379776c27092 P bcd959a07829492db7ded12f1952a44c: Bootstrap starting.
I20260620 20:20:43.072451 2961 tablet_bootstrap.cc:654] T 66e8a939d3cd4596832c379776c27092 P 11b2590223ff4606ba04146e6d544c3e: Neither blocks nor log segments found. Creating new log.
I20260620 20:20:43.073799 2961 log.cc:824] T 66e8a939d3cd4596832c379776c27092 P 11b2590223ff4606ba04146e6d544c3e: Log is configured to *not* fsync() on all Append() calls
I20260620 20:20:43.074712 2962 tablet_bootstrap.cc:654] T 66e8a939d3cd4596832c379776c27092 P cc6b356e5ea445c589807621489a8437: Neither blocks nor log segments found. Creating new log.
I20260620 20:20:43.075248 2963 tablet_bootstrap.cc:654] T 66e8a939d3cd4596832c379776c27092 P bcd959a07829492db7ded12f1952a44c: Neither blocks nor log segments found. Creating new log.
I20260620 20:20:43.075866 2962 log.cc:824] T 66e8a939d3cd4596832c379776c27092 P cc6b356e5ea445c589807621489a8437: Log is configured to *not* fsync() on all Append() calls
I20260620 20:20:43.076961 2963 log.cc:824] T 66e8a939d3cd4596832c379776c27092 P bcd959a07829492db7ded12f1952a44c: Log is configured to *not* fsync() on all Append() calls
I20260620 20:20:43.077260 2961 tablet_bootstrap.cc:492] T 66e8a939d3cd4596832c379776c27092 P 11b2590223ff4606ba04146e6d544c3e: No bootstrap required, opened a new log
I20260620 20:20:43.077607 2961 ts_tablet_manager.cc:1397] T 66e8a939d3cd4596832c379776c27092 P 11b2590223ff4606ba04146e6d544c3e: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260620 20:20:43.079648 2963 tablet_bootstrap.cc:492] T 66e8a939d3cd4596832c379776c27092 P bcd959a07829492db7ded12f1952a44c: No bootstrap required, opened a new log
I20260620 20:20:43.079895 2963 ts_tablet_manager.cc:1397] T 66e8a939d3cd4596832c379776c27092 P bcd959a07829492db7ded12f1952a44c: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260620 20:20:43.083808 2962 tablet_bootstrap.cc:492] T 66e8a939d3cd4596832c379776c27092 P cc6b356e5ea445c589807621489a8437: No bootstrap required, opened a new log
I20260620 20:20:43.084076 2962 ts_tablet_manager.cc:1397] T 66e8a939d3cd4596832c379776c27092 P cc6b356e5ea445c589807621489a8437: Time spent bootstrapping tablet: real 0.014s user 0.007s sys 0.000s
I20260620 20:20:43.084991 2961 raft_consensus.cc:348] T 66e8a939d3cd4596832c379776c27092 P 11b2590223ff4606ba04146e6d544c3e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc6b356e5ea445c589807621489a8437" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37071 } } peers { permanent_uuid: "bcd959a07829492db7ded12f1952a44c" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 44433 } } peers { permanent_uuid: "11b2590223ff4606ba04146e6d544c3e" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 45795 } }
I20260620 20:20:43.085345 2961 raft_consensus.cc:374] T 66e8a939d3cd4596832c379776c27092 P 11b2590223ff4606ba04146e6d544c3e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:20:43.085475 2961 raft_consensus.cc:729] T 66e8a939d3cd4596832c379776c27092 P 11b2590223ff4606ba04146e6d544c3e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 11b2590223ff4606ba04146e6d544c3e, State: Initialized, Role: FOLLOWER
I20260620 20:20:43.086119 2961 consensus_queue.cc:260] T 66e8a939d3cd4596832c379776c27092 P 11b2590223ff4606ba04146e6d544c3e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc6b356e5ea445c589807621489a8437" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37071 } } peers { permanent_uuid: "bcd959a07829492db7ded12f1952a44c" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 44433 } } peers { permanent_uuid: "11b2590223ff4606ba04146e6d544c3e" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 45795 } }
I20260620 20:20:43.089722 2963 raft_consensus.cc:348] T 66e8a939d3cd4596832c379776c27092 P bcd959a07829492db7ded12f1952a44c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc6b356e5ea445c589807621489a8437" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37071 } } peers { permanent_uuid: "bcd959a07829492db7ded12f1952a44c" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 44433 } } peers { permanent_uuid: "11b2590223ff4606ba04146e6d544c3e" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 45795 } }
I20260620 20:20:43.090178 2963 raft_consensus.cc:374] T 66e8a939d3cd4596832c379776c27092 P bcd959a07829492db7ded12f1952a44c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:20:43.090270 2963 raft_consensus.cc:729] T 66e8a939d3cd4596832c379776c27092 P bcd959a07829492db7ded12f1952a44c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bcd959a07829492db7ded12f1952a44c, State: Initialized, Role: FOLLOWER
I20260620 20:20:43.090762 2813 heartbeater.cc:503] Master 127.16.217.62:41191 was elected leader, sending a full tablet report...
I20260620 20:20:43.090945 2963 consensus_queue.cc:260] T 66e8a939d3cd4596832c379776c27092 P bcd959a07829492db7ded12f1952a44c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc6b356e5ea445c589807621489a8437" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37071 } } peers { permanent_uuid: "bcd959a07829492db7ded12f1952a44c" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 44433 } } peers { permanent_uuid: "11b2590223ff4606ba04146e6d544c3e" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 45795 } }
I20260620 20:20:43.091352 2961 ts_tablet_manager.cc:1428] T 66e8a939d3cd4596832c379776c27092 P 11b2590223ff4606ba04146e6d544c3e: Time spent starting tablet: real 0.014s user 0.008s sys 0.007s
I20260620 20:20:43.092330 2943 heartbeater.cc:503] Master 127.16.217.62:41191 was elected leader, sending a full tablet report...
I20260620 20:20:43.092924 2963 ts_tablet_manager.cc:1428] T 66e8a939d3cd4596832c379776c27092 P bcd959a07829492db7ded12f1952a44c: Time spent starting tablet: real 0.013s user 0.013s sys 0.000s
I20260620 20:20:43.094583 2962 raft_consensus.cc:348] T 66e8a939d3cd4596832c379776c27092 P cc6b356e5ea445c589807621489a8437 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc6b356e5ea445c589807621489a8437" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37071 } } peers { permanent_uuid: "bcd959a07829492db7ded12f1952a44c" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 44433 } } peers { permanent_uuid: "11b2590223ff4606ba04146e6d544c3e" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 45795 } }
I20260620 20:20:43.095095 2962 raft_consensus.cc:374] T 66e8a939d3cd4596832c379776c27092 P cc6b356e5ea445c589807621489a8437 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:20:43.095202 2962 raft_consensus.cc:729] T 66e8a939d3cd4596832c379776c27092 P cc6b356e5ea445c589807621489a8437 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc6b356e5ea445c589807621489a8437, State: Initialized, Role: FOLLOWER
I20260620 20:20:43.095750 2962 consensus_queue.cc:260] T 66e8a939d3cd4596832c379776c27092 P cc6b356e5ea445c589807621489a8437 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc6b356e5ea445c589807621489a8437" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37071 } } peers { permanent_uuid: "bcd959a07829492db7ded12f1952a44c" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 44433 } } peers { permanent_uuid: "11b2590223ff4606ba04146e6d544c3e" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 45795 } }
I20260620 20:20:43.097440 2962 ts_tablet_manager.cc:1428] T 66e8a939d3cd4596832c379776c27092 P cc6b356e5ea445c589807621489a8437: Time spent starting tablet: real 0.013s user 0.014s sys 0.000s
W20260620 20:20:43.218415 2944 tablet.cc:2369] T 66e8a939d3cd4596832c379776c27092 P bcd959a07829492db7ded12f1952a44c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 20:20:43.238744 2814 tablet.cc:2369] T 66e8a939d3cd4596832c379776c27092 P 11b2590223ff4606ba04146e6d544c3e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 20:20:43.267838 2683 tablet.cc:2369] T 66e8a939d3cd4596832c379776c27092 P cc6b356e5ea445c589807621489a8437: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 20:20:43.394007 2968 raft_consensus.cc:482] T 66e8a939d3cd4596832c379776c27092 P bcd959a07829492db7ded12f1952a44c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 20:20:43.394335 2968 raft_consensus.cc:504] T 66e8a939d3cd4596832c379776c27092 P bcd959a07829492db7ded12f1952a44c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc6b356e5ea445c589807621489a8437" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37071 } } peers { permanent_uuid: "bcd959a07829492db7ded12f1952a44c" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 44433 } } peers { permanent_uuid: "11b2590223ff4606ba04146e6d544c3e" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 45795 } }
I20260620 20:20:43.396194 2968 leader_election.cc:288] T 66e8a939d3cd4596832c379776c27092 P bcd959a07829492db7ded12f1952a44c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cc6b356e5ea445c589807621489a8437 (127.16.217.1:37071), 11b2590223ff4606ba04146e6d544c3e (127.16.217.2:45795)
I20260620 20:20:43.403101 2955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:43.396618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.217.3:48321 (local address 127.16.217.1:37071)
0620 20:20:43.396834 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0620 20:20:43.396840 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:20:43.397350 (+ 510us) server_negotiation.cc:407] Connection header received
0620 20:20:43.397496 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:20:43.397501 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:20:43.397611 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:20:43.397723 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:20:43.398395 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:43.399088 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:43.401077 (+ 1989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:43.402135 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:43.402284 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:20:43.402366 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0620 20:20:43.402662 (+ 296us) server_negotiation.cc:299] Negotiation successful
0620 20:20:43.402804 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260620 20:20:43.403210 2960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:43.396717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.217.3:39497 (local address 127.16.217.2:45795)
0620 20:20:43.396888 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0620 20:20:43.396896 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 20:20:43.397063 (+ 167us) server_negotiation.cc:407] Connection header received
0620 20:20:43.397374 (+ 311us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:20:43.397379 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:20:43.397460 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:20:43.397539 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:20:43.398199 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:43.398992 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:43.401000 (+ 2008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:43.402176 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:43.402349 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:20:43.402423 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 20:20:43.402773 (+ 350us) server_negotiation.cc:299] Negotiation successful
0620 20:20:43.402923 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 20:20:43.403234 2970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:43.396517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.1:37071 (local address 127.16.217.3:48321)
0620 20:20:43.397222 (+ 705us) negotiation.cc:107] Waiting for socket to connect
0620 20:20:43.397260 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 20:20:43.397377 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:20:43.397727 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:20:43.397731 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:20:43.397764 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:20:43.398185 (+ 421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:20:43.398196 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:20:43.399234 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:20:43.399238 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:20:43.400806 (+ 1568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:20:43.400826 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:20:43.402297 (+ 1471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:20:43.402303 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:20:43.402501 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:20:43.402542 (+ 41us) client_negotiation.cc:769] Sending connection context
0620 20:20:43.402615 (+ 73us) client_negotiation.cc:240] Negotiation successful
0620 20:20:43.402739 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":618,"mutex_wait_us":86,"thread_start_us":429,"threads_started":1}
I20260620 20:20:43.403268 2945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:43.396469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.2:45795 (local address 127.16.217.3:39497)
0620 20:20:43.396929 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0620 20:20:43.396955 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0620 20:20:43.397167 (+ 212us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:20:43.397624 (+ 457us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:20:43.397629 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:20:43.397683 (+ 54us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:20:43.398031 (+ 348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:20:43.398040 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:20:43.399144 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:20:43.399153 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:20:43.400800 (+ 1647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:20:43.400842 (+ 42us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:20:43.402335 (+ 1493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:20:43.402341 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:20:43.402586 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:20:43.402634 (+ 48us) client_negotiation.cc:769] Sending connection context
0620 20:20:43.402733 (+ 99us) client_negotiation.cc:240] Negotiation successful
0620 20:20:43.402872 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":217,"mutex_wait_us":127}
I20260620 20:20:43.404276 2637 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66e8a939d3cd4596832c379776c27092" candidate_uuid: "bcd959a07829492db7ded12f1952a44c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc6b356e5ea445c589807621489a8437" is_pre_election: true
I20260620 20:20:43.404650 2768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66e8a939d3cd4596832c379776c27092" candidate_uuid: "bcd959a07829492db7ded12f1952a44c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "11b2590223ff4606ba04146e6d544c3e" is_pre_election: true
I20260620 20:20:43.404914 2637 raft_consensus.cc:2454] T 66e8a939d3cd4596832c379776c27092 P cc6b356e5ea445c589807621489a8437 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bcd959a07829492db7ded12f1952a44c in term 0.
I20260620 20:20:43.405233 2768 raft_consensus.cc:2454] T 66e8a939d3cd4596832c379776c27092 P 11b2590223ff4606ba04146e6d544c3e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bcd959a07829492db7ded12f1952a44c in term 0.
I20260620 20:20:43.405676 2831 leader_election.cc:302] T 66e8a939d3cd4596832c379776c27092 P bcd959a07829492db7ded12f1952a44c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bcd959a07829492db7ded12f1952a44c, cc6b356e5ea445c589807621489a8437; no voters:
I20260620 20:20:43.406141 2968 raft_consensus.cc:2781] T 66e8a939d3cd4596832c379776c27092 P bcd959a07829492db7ded12f1952a44c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 20:20:43.406302 2968 raft_consensus.cc:482] T 66e8a939d3cd4596832c379776c27092 P bcd959a07829492db7ded12f1952a44c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 20:20:43.406409 2968 raft_consensus.cc:3037] T 66e8a939d3cd4596832c379776c27092 P bcd959a07829492db7ded12f1952a44c [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:20:43.408619 2968 raft_consensus.cc:504] T 66e8a939d3cd4596832c379776c27092 P bcd959a07829492db7ded12f1952a44c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc6b356e5ea445c589807621489a8437" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37071 } } peers { permanent_uuid: "bcd959a07829492db7ded12f1952a44c" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 44433 } } peers { permanent_uuid: "11b2590223ff4606ba04146e6d544c3e" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 45795 } }
I20260620 20:20:43.409235 2968 leader_election.cc:288] T 66e8a939d3cd4596832c379776c27092 P bcd959a07829492db7ded12f1952a44c [CANDIDATE]: Term 1 election: Requested vote from peers cc6b356e5ea445c589807621489a8437 (127.16.217.1:37071), 11b2590223ff4606ba04146e6d544c3e (127.16.217.2:45795)
I20260620 20:20:43.409659 2637 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66e8a939d3cd4596832c379776c27092" candidate_uuid: "bcd959a07829492db7ded12f1952a44c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc6b356e5ea445c589807621489a8437"
I20260620 20:20:43.409698 2768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66e8a939d3cd4596832c379776c27092" candidate_uuid: "bcd959a07829492db7ded12f1952a44c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "11b2590223ff4606ba04146e6d544c3e"
I20260620 20:20:43.409852 2637 raft_consensus.cc:3037] T 66e8a939d3cd4596832c379776c27092 P cc6b356e5ea445c589807621489a8437 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:20:43.409962 2768 raft_consensus.cc:3037] T 66e8a939d3cd4596832c379776c27092 P 11b2590223ff4606ba04146e6d544c3e [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:20:43.412149 2637 raft_consensus.cc:2454] T 66e8a939d3cd4596832c379776c27092 P cc6b356e5ea445c589807621489a8437 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bcd959a07829492db7ded12f1952a44c in term 1.
I20260620 20:20:43.412173 2768 raft_consensus.cc:2454] T 66e8a939d3cd4596832c379776c27092 P 11b2590223ff4606ba04146e6d544c3e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bcd959a07829492db7ded12f1952a44c in term 1.
I20260620 20:20:43.412555 2832 leader_election.cc:302] T 66e8a939d3cd4596832c379776c27092 P bcd959a07829492db7ded12f1952a44c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 11b2590223ff4606ba04146e6d544c3e, bcd959a07829492db7ded12f1952a44c; no voters:
I20260620 20:20:43.412851 2968 raft_consensus.cc:2781] T 66e8a939d3cd4596832c379776c27092 P bcd959a07829492db7ded12f1952a44c [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:20:43.413295 2968 raft_consensus.cc:686] T 66e8a939d3cd4596832c379776c27092 P bcd959a07829492db7ded12f1952a44c [term 1 LEADER]: Becoming Leader. State: Replica: bcd959a07829492db7ded12f1952a44c, State: Running, Role: LEADER
I20260620 20:20:43.413813 2968 consensus_queue.cc:237] T 66e8a939d3cd4596832c379776c27092 P bcd959a07829492db7ded12f1952a44c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc6b356e5ea445c589807621489a8437" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37071 } } peers { permanent_uuid: "bcd959a07829492db7ded12f1952a44c" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 44433 } } peers { permanent_uuid: "11b2590223ff4606ba04146e6d544c3e" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 45795 } }
I20260620 20:20:43.418607 2496 catalog_manager.cc:5495] T 66e8a939d3cd4596832c379776c27092 P bcd959a07829492db7ded12f1952a44c reported cstate change: term changed from 0 to 1, leader changed from <none> to bcd959a07829492db7ded12f1952a44c (127.16.217.3). New cstate: current_term: 1 leader_uuid: "bcd959a07829492db7ded12f1952a44c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc6b356e5ea445c589807621489a8437" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37071 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bcd959a07829492db7ded12f1952a44c" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 44433 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "11b2590223ff4606ba04146e6d544c3e" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 45795 } health_report { overall_health: UNKNOWN } } }
I20260620 20:20:43.502862 2959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:43.497116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43792 (local address 127.16.217.3:44433)
0620 20:20:43.497338 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0620 20:20:43.497343 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:20:43.497402 (+ 59us) server_negotiation.cc:407] Connection header received
0620 20:20:43.497646 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:20:43.497650 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:20:43.497734 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:20:43.497849 (+ 115us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 20:20:43.498252 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:43.498982 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:43.500456 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:43.500892 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:43.500996 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:20:43.501553 (+ 557us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 20:20:43.502213 (+ 660us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 20:20:43.502379 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0620 20:20:43.502466 (+ 87us) server_negotiation.cc:299] Negotiation successful
0620 20:20:43.502573 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260620 20:20:43.510643 2768 raft_consensus.cc:1261] T 66e8a939d3cd4596832c379776c27092 P 11b2590223ff4606ba04146e6d544c3e [term 1 FOLLOWER]: Refusing update from remote peer bcd959a07829492db7ded12f1952a44c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:20:43.510700 2637 raft_consensus.cc:1261] T 66e8a939d3cd4596832c379776c27092 P cc6b356e5ea445c589807621489a8437 [term 1 FOLLOWER]: Refusing update from remote peer bcd959a07829492db7ded12f1952a44c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:20:43.511588 2968 consensus_queue.cc:1035] T 66e8a939d3cd4596832c379776c27092 P bcd959a07829492db7ded12f1952a44c [LEADER]: Connected to new peer: Peer: permanent_uuid: "11b2590223ff4606ba04146e6d544c3e" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 45795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:20:43.511860 2971 consensus_queue.cc:1035] T 66e8a939d3cd4596832c379776c27092 P bcd959a07829492db7ded12f1952a44c [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc6b356e5ea445c589807621489a8437" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:20:43.526055 2974 mvcc.cc:205] Tried to move back new op lower bound from 7299018111010488320 to 7299018110627356672. Current Snapshot: MvccSnapshot[applied={T|T < 7299018111010488320}]
I20260620 20:20:43.539098 2496 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44224:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 20:20:43.539386 2496 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44224:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 20:20:43.543550 2496 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f576a95bc9ee4f509de5fbac6e6d9758: Sending DeleteTablet for 3 replicas of tablet 66e8a939d3cd4596832c379776c27092
I20260620 20:20:43.544610 2617 tablet_service.cc:1514] Processing DeleteTablet for tablet 66e8a939d3cd4596832c379776c27092 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:20:43 UTC) from {username='slave'} at 127.0.0.1:37560
I20260620 20:20:43.548099 2878 tablet_service.cc:1514] Processing DeleteTablet for tablet 66e8a939d3cd4596832c379776c27092 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:20:43 UTC) from {username='slave'} at 127.0.0.1:43788
I20260620 20:20:43.548514 2984 tablet_replica.cc:330] T 66e8a939d3cd4596832c379776c27092 P cc6b356e5ea445c589807621489a8437: stopping tablet replica
I20260620 20:20:43.548693 2748 tablet_service.cc:1514] Processing DeleteTablet for tablet 66e8a939d3cd4596832c379776c27092 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:20:43 UTC) from {username='slave'} at 127.0.0.1:53148
I20260620 20:20:43.549012 2984 raft_consensus.cc:2229] T 66e8a939d3cd4596832c379776c27092 P cc6b356e5ea445c589807621489a8437 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 20:20:43.549588 2984 raft_consensus.cc:2258] T 66e8a939d3cd4596832c379776c27092 P cc6b356e5ea445c589807621489a8437 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:20:43.552016 2984 ts_tablet_manager.cc:1905] T 66e8a939d3cd4596832c379776c27092 P cc6b356e5ea445c589807621489a8437: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:20:43.552018 2747 tablet_service.cc:1514] Processing DeleteTablet for tablet 66e8a939d3cd4596832c379776c27092 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:20:43 UTC) from {username='slave'} at 127.0.0.1:53148
I20260620 20:20:43.553820 17252 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu pid 2555
I20260620 20:20:43.559419 2985 tablet_replica.cc:330] T 66e8a939d3cd4596832c379776c27092 P bcd959a07829492db7ded12f1952a44c: stopping tablet replica
I20260620 20:20:43.559827 2985 raft_consensus.cc:2229] T 66e8a939d3cd4596832c379776c27092 P bcd959a07829492db7ded12f1952a44c [term 1 LEADER]: Raft consensus shutting down.
I20260620 20:20:43.560014 2986 tablet_replica.cc:330] T 66e8a939d3cd4596832c379776c27092 P 11b2590223ff4606ba04146e6d544c3e: stopping tablet replica
I20260620 20:20:43.560429 2984 ts_tablet_manager.cc:1918] T 66e8a939d3cd4596832c379776c27092 P cc6b356e5ea445c589807621489a8437: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 20:20:43.560609 2984 log.cc:1196] T 66e8a939d3cd4596832c379776c27092 P cc6b356e5ea445c589807621489a8437: Deleting WAL directory at /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986536687152-17252-0/minicluster-data/ts-0/wal/wals/66e8a939d3cd4596832c379776c27092
I20260620 20:20:43.560624 2986 raft_consensus.cc:2229] T 66e8a939d3cd4596832c379776c27092 P 11b2590223ff4606ba04146e6d544c3e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 20:20:43.561107 2984 ts_tablet_manager.cc:1939] T 66e8a939d3cd4596832c379776c27092 P cc6b356e5ea445c589807621489a8437: Deleting consensus metadata
I20260620 20:20:43.561180 2986 raft_consensus.cc:2258] T 66e8a939d3cd4596832c379776c27092 P 11b2590223ff4606ba04146e6d544c3e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:20:43.562582 2985 raft_consensus.cc:2258] T 66e8a939d3cd4596832c379776c27092 P bcd959a07829492db7ded12f1952a44c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:20:43.563586 2986 ts_tablet_manager.cc:1905] T 66e8a939d3cd4596832c379776c27092 P 11b2590223ff4606ba04146e6d544c3e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:20:43.564688 2985 ts_tablet_manager.cc:1905] T 66e8a939d3cd4596832c379776c27092 P bcd959a07829492db7ded12f1952a44c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:20:43.570654 2986 ts_tablet_manager.cc:1918] T 66e8a939d3cd4596832c379776c27092 P 11b2590223ff4606ba04146e6d544c3e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 20:20:43.570827 2986 log.cc:1196] T 66e8a939d3cd4596832c379776c27092 P 11b2590223ff4606ba04146e6d544c3e: Deleting WAL directory at /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986536687152-17252-0/minicluster-data/ts-1/wal/wals/66e8a939d3cd4596832c379776c27092
I20260620 20:20:43.571348 2986 ts_tablet_manager.cc:1939] T 66e8a939d3cd4596832c379776c27092 P 11b2590223ff4606ba04146e6d544c3e: Deleting consensus metadata
I20260620 20:20:43.571790 2480 catalog_manager.cc:4841] TS cc6b356e5ea445c589807621489a8437 (127.16.217.1:37071): tablet 66e8a939d3cd4596832c379776c27092 (table test-table [id=325b285d855f4ac198d2b6bbabc601f9]) successfully deleted
I20260620 20:20:43.573115 2481 catalog_manager.cc:4841] TS 11b2590223ff4606ba04146e6d544c3e (127.16.217.2:45795): tablet 66e8a939d3cd4596832c379776c27092 (table test-table [id=325b285d855f4ac198d2b6bbabc601f9]) successfully deleted
I20260620 20:20:43.573578 2955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:43.555371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37570 (local address 127.16.217.1:37071)
0620 20:20:43.555551 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0620 20:20:43.555558 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:20:43.565128 (+ 9570us) server_negotiation.cc:407] Connection header received
0620 20:20:43.565618 (+ 490us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:20:43.565624 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:20:43.565740 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:20:43.565819 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:20:43.566490 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:43.567562 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:43.569580 (+ 2018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:43.570220 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:43.570320 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:20:43.571915 (+ 1595us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:20:43.571926 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:20:43.571942 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:20:43.572713 (+ 771us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:20:43.572780 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:20:43.572804 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:20:43.572886 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0620 20:20:43.573186 (+ 300us) server_negotiation.cc:299] Negotiation successful
0620 20:20:43.573293 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
W20260620 20:20:43.573688 2481 catalog_manager.cc:4808] TS 11b2590223ff4606ba04146e6d544c3e (127.16.217.2:45795): delete failed for tablet 66e8a939d3cd4596832c379776c27092 because the tablet was not found. No further retry: Not found: Tablet not found: 66e8a939d3cd4596832c379776c27092
I20260620 20:20:43.574111 2985 ts_tablet_manager.cc:1918] T 66e8a939d3cd4596832c379776c27092 P bcd959a07829492db7ded12f1952a44c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 20:20:43.574276 2985 log.cc:1196] T 66e8a939d3cd4596832c379776c27092 P bcd959a07829492db7ded12f1952a44c: Deleting WAL directory at /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986536687152-17252-0/minicluster-data/ts-2/wal/wals/66e8a939d3cd4596832c379776c27092
I20260620 20:20:43.574817 2677 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:20:43.574816 2985 ts_tablet_manager.cc:1939] T 66e8a939d3cd4596832c379776c27092 P bcd959a07829492db7ded12f1952a44c: Deleting consensus metadata
I20260620 20:20:43.576694 2483 catalog_manager.cc:4841] TS bcd959a07829492db7ded12f1952a44c (127.16.217.3:44433): tablet 66e8a939d3cd4596832c379776c27092 (table test-table [id=325b285d855f4ac198d2b6bbabc601f9]) successfully deleted
I20260620 20:20:43.665071 17252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu with pid 2555
I20260620 20:20:43.683136 17252 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu pid 2686
I20260620 20:20:43.690618 2960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:43.683961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53162 (local address 127.16.217.2:45795)
0620 20:20:43.684141 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0620 20:20:43.684147 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:20:43.684167 (+ 20us) server_negotiation.cc:407] Connection header received
0620 20:20:43.684367 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:20:43.684373 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:20:43.684466 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:20:43.684584 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:20:43.684956 (+ 372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:43.685699 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:43.687086 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:43.687776 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:43.687918 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:20:43.688949 (+ 1031us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:20:43.688961 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:20:43.688982 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:20:43.689796 (+ 814us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:20:43.689859 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:20:43.689920 (+ 61us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:20:43.690041 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0620 20:20:43.690119 (+ 78us) server_negotiation.cc:299] Negotiation successful
0620 20:20:43.690233 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 20:20:43.691357 2808 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:20:43.778260 17252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu with pid 2686
I20260620 20:20:43.795831 17252 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu pid 2816
I20260620 20:20:43.803393 2959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:43.796631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43800 (local address 127.16.217.3:44433)
0620 20:20:43.796833 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0620 20:20:43.796841 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 20:20:43.796906 (+ 65us) server_negotiation.cc:407] Connection header received
0620 20:20:43.797110 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:20:43.797115 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:20:43.797206 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:20:43.797320 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:20:43.797736 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:43.798611 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:43.799960 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:43.800412 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:43.800544 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:20:43.801756 (+ 1212us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:20:43.801767 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:20:43.801786 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:20:43.802593 (+ 807us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:20:43.802654 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:20:43.802679 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:20:43.802810 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0620 20:20:43.802911 (+ 101us) server_negotiation.cc:299] Negotiation successful
0620 20:20:43.803025 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
I20260620 20:20:43.804177 2938 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:20:43.924314 17252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu with pid 2816
I20260620 20:20:43.939883 17252 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu pid 2465
I20260620 20:20:43.940934 2526 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:20:44.050496 17252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu with pid 2465
2026-06-20T20:20:44Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (3115 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986536687152-17252-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 20:20:44 dist-test-slave-2hpg krb5kdc[2994](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 20:20:44 dist-test-slave-2hpg krb5kdc[2994](info): set up 2 sockets
Jun 20 20:20:44 dist-test-slave-2hpg krb5kdc[2994](info): commencing operation
krb5kdc: starting...
W20260620 20:20:45.162432 17252 mini_kdc.cc:121] Time spent starting KDC: real 1.071s user 0.003s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986536687152-17252-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986536687152-17252-0/krb5kdc/test-user.keytab.
Jun 20 20:20:45 dist-test-slave-2hpg krb5kdc[2994](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986845, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T20:20:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T20:20:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.217.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.217.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.217.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.62.keytab.
Entry for principal kudu/127.16.217.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.62.keytab.
I20260620 20:20:45.410848 17252 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
/tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986536687152-17252-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986536687152-17252-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986536687152-17252-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986536687152-17252-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.217.62:42287
--webserver_interface=127.16.217.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.62.keytab
--principal=kudu/127.16.217.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.217.20:42511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.217.62:42287
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986536687152-17252-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986536687152-17252-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986536687152-17252-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:20:45.737488 3007 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:20:45.737766 3007 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:20:45.737937 3007 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:20:45.745013 3007 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:20:45.745108 3007 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:20:45.745210 3007 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:20:45.745307 3007 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:20:45.745385 3007 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:20:45.755055 3007 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.217.20:42511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986536687152-17252-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986536687152-17252-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.217.62:42287
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.62.keytab
--principal=kudu/127.16.217.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.217.62:42287
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986536687152-17252-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.217.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986536687152-17252-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:20:45.756299 3007 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:20:45.757592 3007 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:20:45.769198 3012 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:20:45.769230 3013 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:20:45.769253 3015 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:20:45.770095 3007 server_base.cc:1034] running on GCE node
I20260620 20:20:45.770879 3007 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:20:45.772212 3007 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:20:45.773442 3007 hybrid_clock.cc:648] HybridClock initialized: now 1781986845773401 us; error 47 us; skew 500 ppm
I20260620 20:20:45.773798 3007 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:20:45 dist-test-slave-2hpg krb5kdc[2994](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986845, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.217.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:20:45.776888 3007 init.cc:377] Logged in from keytab as kudu/127.16.217.62@KRBTEST.COM (short username kudu)
I20260620 20:20:45.778424 3007 webserver.cc:460] Webserver started at http://127.16.217.62:41129/ using document root <none> and password file <none>
I20260620 20:20:45.779095 3007 fs_manager.cc:359] Metadata directory not provided
I20260620 20:20:45.779239 3007 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:20:45.779532 3007 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:20:45.782512 3007 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986536687152-17252-0/minicluster-data/master-0/data/instance:
uuid: "d05bde36c6ec4fb4ad274504ad322e9e"
format_stamp: "Formatted at 2026-06-20 20:20:45 on dist-test-slave-2hpg"
server_key: "7bb4e815fe186f9a567d7b49e7925789"
server_key_iv: "e0852ddd2f353776f4feb63db8a5966a"
server_key_version: "encryptionkey@0"
I20260620 20:20:45.783260 3007 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986536687152-17252-0/minicluster-data/master-0/wal/instance:
uuid: "d05bde36c6ec4fb4ad274504ad322e9e"
format_stamp: "Formatted at 2026-06-20 20:20:45 on dist-test-slave-2hpg"
server_key: "7bb4e815fe186f9a567d7b49e7925789"
server_key_iv: "e0852ddd2f353776f4feb63db8a5966a"
server_key_version: "encryptionkey@0"
I20260620 20:20:45.788450 3007 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260620 20:20:45.791916 3022 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:20:45.793278 3007 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 20:20:45.793459 3007 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986536687152-17252-0/minicluster-data/master-0/data,/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986536687152-17252-0/minicluster-data/master-0/wal
uuid: "d05bde36c6ec4fb4ad274504ad322e9e"
format_stamp: "Formatted at 2026-06-20 20:20:45 on dist-test-slave-2hpg"
server_key: "7bb4e815fe186f9a567d7b49e7925789"
server_key_iv: "e0852ddd2f353776f4feb63db8a5966a"
server_key_version: "encryptionkey@0"
I20260620 20:20:45.793651 3007 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986536687152-17252-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986536687152-17252-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986536687152-17252-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:20:45.809625 3007 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:20:45.814126 3007 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:20:45.814404 3007 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:20:45.833352 3007 rpc_server.cc:307] RPC server started. Bound to: 127.16.217.62:42287
I20260620 20:20:45.833377 3073 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.217.62:42287 every 8 connection(s)
I20260620 20:20:45.835220 3007 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986536687152-17252-0/minicluster-data/master-0/data/info.pb
I20260620 20:20:45.838915 3074 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:20:45.842618 17252 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu as pid 3007
I20260620 20:20:45.842809 17252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986536687152-17252-0/minicluster-data/master-0/wal/instance
I20260620 20:20:45.843222 17252 external_mini_cluster.cc:1428] Setting key 519ec23fd43245b07c575163cdb87da3
I20260620 20:20:45.851425 3074 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d05bde36c6ec4fb4ad274504ad322e9e: Bootstrap starting.
I20260620 20:20:45.855091 3074 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d05bde36c6ec4fb4ad274504ad322e9e: Neither blocks nor log segments found. Creating new log.
I20260620 20:20:45.856375 3074 log.cc:824] T 00000000000000000000000000000000 P d05bde36c6ec4fb4ad274504ad322e9e: Log is configured to *not* fsync() on all Append() calls
Jun 20 20:20:45 dist-test-slave-2hpg krb5kdc[2994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986845, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.217.62@KRBTEST.COM
I20260620 20:20:45.859467 3074 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d05bde36c6ec4fb4ad274504ad322e9e: No bootstrap required, opened a new log
I20260620 20:20:45.867982 3077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:45.844927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34706 (local address 127.16.217.62:42287)
0620 20:20:45.845663 (+ 736us) server_negotiation.cc:206] Beginning negotiation
0620 20:20:45.845674 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 20:20:45.845722 (+ 48us) server_negotiation.cc:407] Connection header received
0620 20:20:45.847362 (+ 1640us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:20:45.847383 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:20:45.847765 (+ 382us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:20:45.848063 (+ 298us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:20:45.849946 (+ 1883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:45.851251 (+ 1305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:45.853113 (+ 1862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:45.853841 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:45.854054 (+ 213us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:20:45.858754 (+ 4700us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:20:45.858764 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:20:45.858784 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:20:45.858833 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:20:45.863132 (+ 4299us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:20:45.864003 (+ 871us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:20:45.864012 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:20:45.864022 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:20:45.864255 (+ 233us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:20:45.864671 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:20:45.864676 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:20:45.864679 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:20:45.865421 (+ 742us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:20:45.865736 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0620 20:20:45.866151 (+ 415us) server_negotiation.cc:299] Negotiation successful
0620 20:20:45.866599 (+ 448us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":549,"thread_start_us":356,"threads_started":1}
I20260620 20:20:45.870476 3074 raft_consensus.cc:348] T 00000000000000000000000000000000 P d05bde36c6ec4fb4ad274504ad322e9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d05bde36c6ec4fb4ad274504ad322e9e" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 42287 } }
I20260620 20:20:45.870903 3074 raft_consensus.cc:374] T 00000000000000000000000000000000 P d05bde36c6ec4fb4ad274504ad322e9e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:20:45.871014 3074 raft_consensus.cc:729] T 00000000000000000000000000000000 P d05bde36c6ec4fb4ad274504ad322e9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d05bde36c6ec4fb4ad274504ad322e9e, State: Initialized, Role: FOLLOWER
I20260620 20:20:45.871583 3074 consensus_queue.cc:260] T 00000000000000000000000000000000 P d05bde36c6ec4fb4ad274504ad322e9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d05bde36c6ec4fb4ad274504ad322e9e" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 42287 } }
I20260620 20:20:45.871822 3074 raft_consensus.cc:388] T 00000000000000000000000000000000 P d05bde36c6ec4fb4ad274504ad322e9e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 20:20:45.871959 3074 raft_consensus.cc:482] T 00000000000000000000000000000000 P d05bde36c6ec4fb4ad274504ad322e9e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 20:20:45.872125 3074 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d05bde36c6ec4fb4ad274504ad322e9e [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:20:45.874668 3074 raft_consensus.cc:504] T 00000000000000000000000000000000 P d05bde36c6ec4fb4ad274504ad322e9e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d05bde36c6ec4fb4ad274504ad322e9e" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 42287 } }
I20260620 20:20:45.875180 3074 leader_election.cc:302] T 00000000000000000000000000000000 P d05bde36c6ec4fb4ad274504ad322e9e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d05bde36c6ec4fb4ad274504ad322e9e; no voters:
I20260620 20:20:45.875808 3074 leader_election.cc:288] T 00000000000000000000000000000000 P d05bde36c6ec4fb4ad274504ad322e9e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 20:20:45.875962 3079 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d05bde36c6ec4fb4ad274504ad322e9e [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:20:45.877077 3079 raft_consensus.cc:686] T 00000000000000000000000000000000 P d05bde36c6ec4fb4ad274504ad322e9e [term 1 LEADER]: Becoming Leader. State: Replica: d05bde36c6ec4fb4ad274504ad322e9e, State: Running, Role: LEADER
I20260620 20:20:45.877591 3079 consensus_queue.cc:237] T 00000000000000000000000000000000 P d05bde36c6ec4fb4ad274504ad322e9e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d05bde36c6ec4fb4ad274504ad322e9e" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 42287 } }
I20260620 20:20:45.877919 3074 sys_catalog.cc:564] T 00000000000000000000000000000000 P d05bde36c6ec4fb4ad274504ad322e9e [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:20:45.880811 3080 sys_catalog.cc:455] T 00000000000000000000000000000000 P d05bde36c6ec4fb4ad274504ad322e9e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d05bde36c6ec4fb4ad274504ad322e9e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d05bde36c6ec4fb4ad274504ad322e9e" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 42287 } } }
I20260620 20:20:45.880789 3081 sys_catalog.cc:455] T 00000000000000000000000000000000 P d05bde36c6ec4fb4ad274504ad322e9e [sys.catalog]: SysCatalogTable state changed. Reason: New leader d05bde36c6ec4fb4ad274504ad322e9e. Latest consensus state: current_term: 1 leader_uuid: "d05bde36c6ec4fb4ad274504ad322e9e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d05bde36c6ec4fb4ad274504ad322e9e" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 42287 } } }
I20260620 20:20:45.881217 3080 sys_catalog.cc:458] T 00000000000000000000000000000000 P d05bde36c6ec4fb4ad274504ad322e9e [sys.catalog]: This master's current role is: LEADER
I20260620 20:20:45.881261 3081 sys_catalog.cc:458] T 00000000000000000000000000000000 P d05bde36c6ec4fb4ad274504ad322e9e [sys.catalog]: This master's current role is: LEADER
I20260620 20:20:45.883157 3085 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 20:20:45.889858 3085 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 20:20:45.899376 3085 catalog_manager.cc:1347] Generated new cluster ID: 55971096705f41eeb200dfef76d82782
I20260620 20:20:45.899498 3085 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 20:20:45.918718 3085 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 20:20:45.920703 3085 catalog_manager.cc:1504] Loading token signing keys...
I20260620 20:20:45.931680 3085 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d05bde36c6ec4fb4ad274504ad322e9e: Generated new TSK 0
I20260620 20:20:45.932734 3085 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.217.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.217.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.217.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.1.keytab.
Entry for principal kudu/127.16.217.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.1.keytab.
I20260620 20:20:46.006877 17252 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
/tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986536687152-17252-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986536687152-17252-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986536687152-17252-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986536687152-17252-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.217.1:0
--local_ip_for_outbound_sockets=127.16.217.1
--webserver_interface=127.16.217.1
--webserver_port=0
--tserver_master_addrs=127.16.217.62:42287
--keytab_file=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.1.keytab
--principal=kudu/127.16.217.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.217.20:42511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986536687152-17252-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986536687152-17252-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986536687152-17252-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:20:46.361542 3100 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:20:46.361845 3100 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:20:46.362041 3100 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:20:46.369231 3100 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:20:46.369361 3100 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:20:46.369526 3100 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.217.1
I20260620 20:20:46.379310 3100 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.217.20:42511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986536687152-17252-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986536687152-17252-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.1.keytab
--principal=kudu/127.16.217.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.217.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986536687152-17252-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.217.1
--webserver_port=0
--tserver_master_addrs=127.16.217.62:42287
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.217.1
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986536687152-17252-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:20:46.380581 3100 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:20:46.381914 3100 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:20:46.396116 3106 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:20:46.396116 3108 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:20:46.396221 3105 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:20:46.397441 3100 server_base.cc:1034] running on GCE node
I20260620 20:20:46.398141 3100 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:20:46.399098 3100 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:20:46.400324 3100 hybrid_clock.cc:648] HybridClock initialized: now 1781986846400275 us; error 42 us; skew 500 ppm
I20260620 20:20:46.400647 3100 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:20:46 dist-test-slave-2hpg krb5kdc[2994](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986846, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.217.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:20:46.408552 3100 init.cc:377] Logged in from keytab as kudu/127.16.217.1@KRBTEST.COM (short username kudu)
I20260620 20:20:46.409992 3100 webserver.cc:460] Webserver started at http://127.16.217.1:36399/ using document root <none> and password file <none>
I20260620 20:20:46.410673 3100 fs_manager.cc:359] Metadata directory not provided
I20260620 20:20:46.410789 3100 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:20:46.411064 3100 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:20:46.414155 3100 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986536687152-17252-0/minicluster-data/ts-0/data/instance:
uuid: "8134f7d52e5a4fbfbe15fe2e0bce6dc1"
format_stamp: "Formatted at 2026-06-20 20:20:46 on dist-test-slave-2hpg"
server_key: "6f74e884ece047e9cc7d12f91b0b441a"
server_key_iv: "864e53cec64b030fd84d266762a650e4"
server_key_version: "encryptionkey@0"
I20260620 20:20:46.414896 3100 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986536687152-17252-0/minicluster-data/ts-0/wal/instance:
uuid: "8134f7d52e5a4fbfbe15fe2e0bce6dc1"
format_stamp: "Formatted at 2026-06-20 20:20:46 on dist-test-slave-2hpg"
server_key: "6f74e884ece047e9cc7d12f91b0b441a"
server_key_iv: "864e53cec64b030fd84d266762a650e4"
server_key_version: "encryptionkey@0"
I20260620 20:20:46.419945 3100 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260620 20:20:46.423314 3115 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:20:46.424629 3100 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 20:20:46.424758 3100 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986536687152-17252-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986536687152-17252-0/minicluster-data/ts-0/wal
uuid: "8134f7d52e5a4fbfbe15fe2e0bce6dc1"
format_stamp: "Formatted at 2026-06-20 20:20:46 on dist-test-slave-2hpg"
server_key: "6f74e884ece047e9cc7d12f91b0b441a"
server_key_iv: "864e53cec64b030fd84d266762a650e4"
server_key_version: "encryptionkey@0"
I20260620 20:20:46.424930 3100 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986536687152-17252-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986536687152-17252-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986536687152-17252-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:20:46.446101 3100 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:20:46.450378 3100 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:20:46.450616 3100 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:20:46.451550 3100 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:20:46.452797 3100 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:20:46.452872 3100 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:20:46.452961 3100 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:20:46.453006 3100 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:20:46.488245 3100 rpc_server.cc:307] RPC server started. Bound to: 127.16.217.1:35609
I20260620 20:20:46.488287 3227 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.217.1:35609 every 8 connection(s)
I20260620 20:20:46.489614 3100 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986536687152-17252-0/minicluster-data/ts-0/data/info.pb
I20260620 20:20:46.494326 17252 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu as pid 3100
I20260620 20:20:46.494510 17252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986536687152-17252-0/minicluster-data/ts-0/wal/instance
I20260620 20:20:46.494913 17252 external_mini_cluster.cc:1428] Setting key 455ec2aec6ca6dc3e65738d331216e30
Jun 20 20:20:46 dist-test-slave-2hpg krb5kdc[2994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986846, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.217.1@KRBTEST.COM for kudu/127.16.217.62@KRBTEST.COM
I20260620 20:20:46.509433 3230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:46.492222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.217.1:60543 (local address 127.16.217.62:42287)
0620 20:20:46.492672 (+ 450us) server_negotiation.cc:206] Beginning negotiation
0620 20:20:46.492679 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:20:46.493646 (+ 967us) server_negotiation.cc:407] Connection header received
0620 20:20:46.495664 (+ 2018us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:20:46.495669 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:20:46.495760 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:20:46.495875 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:20:46.497497 (+ 1622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:46.498350 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:46.499845 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:46.500250 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:46.500399 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:20:46.503790 (+ 3391us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:20:46.503795 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:20:46.503798 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:20:46.503831 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:20:46.506457 (+ 2626us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:20:46.507176 (+ 719us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:20:46.507187 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:20:46.507189 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:20:46.507312 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:20:46.507728 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:20:46.507734 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:20:46.507740 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:20:46.508000 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:20:46.508148 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0620 20:20:46.508823 (+ 675us) server_negotiation.cc:299] Negotiation successful
0620 20:20:46.509046 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":322,"thread_start_us":192,"threads_started":1}
I20260620 20:20:46.510464 3231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:46.492795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:42287 (local address 127.16.217.1:60543)
0620 20:20:46.493487 (+ 692us) negotiation.cc:107] Waiting for socket to connect
0620 20:20:46.493528 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 20:20:46.495375 (+ 1847us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:20:46.496149 (+ 774us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:20:46.496162 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:20:46.496744 (+ 582us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:20:46.497336 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:20:46.497355 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:20:46.498493 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:20:46.498498 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:20:46.499695 (+ 1197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:20:46.499707 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:20:46.500466 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:20:46.500475 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:20:46.500726 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:20:46.501428 (+ 702us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:20:46.501449 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:20:46.503576 (+ 2127us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:20:46.506603 (+ 3027us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:20:46.506612 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:20:46.506627 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:20:46.507051 (+ 424us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:20:46.507429 (+ 378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:20:46.507434 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:20:46.507437 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:20:46.507622 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:20:46.508132 (+ 510us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:20:46.508140 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:20:46.508466 (+ 326us) client_negotiation.cc:769] Sending connection context
0620 20:20:46.508739 (+ 273us) client_negotiation.cc:240] Negotiation successful
0620 20:20:46.509274 (+ 535us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":469,"thread_start_us":273,"threads_started":1}
I20260620 20:20:46.512006 3228 heartbeater.cc:347] Connected to a master server at 127.16.217.62:42287
I20260620 20:20:46.512328 3228 heartbeater.cc:464] Registering TS with master...
I20260620 20:20:46.513176 3228 heartbeater.cc:511] Master 127.16.217.62:42287 requested a full tablet report, sending...
I20260620 20:20:46.515338 3039 ts_manager.cc:194] Registered new tserver with Master: 8134f7d52e5a4fbfbe15fe2e0bce6dc1 (127.16.217.1:35609)
I20260620 20:20:46.516826 3039 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.217.1@KRBTEST.COM'} at 127.16.217.1:60543
WARNING: no policy specified for kudu/127.16.217.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.217.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.217.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.2.keytab.
Entry for principal kudu/127.16.217.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.2.keytab.
I20260620 20:20:46.549547 17252 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
/tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986536687152-17252-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986536687152-17252-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986536687152-17252-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986536687152-17252-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.217.2:0
--local_ip_for_outbound_sockets=127.16.217.2
--webserver_interface=127.16.217.2
--webserver_port=0
--tserver_master_addrs=127.16.217.62:42287
--keytab_file=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.2.keytab
--principal=kudu/127.16.217.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.217.20:42511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986536687152-17252-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986536687152-17252-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986536687152-17252-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:20:46.872303 3234 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:20:46.872570 3234 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:20:46.872660 3234 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:20:46.879747 3234 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:20:46.879851 3234 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:20:46.879962 3234 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.217.2
I20260620 20:20:46.889557 3234 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.217.20:42511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986536687152-17252-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986536687152-17252-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.2.keytab
--principal=kudu/127.16.217.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.217.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986536687152-17252-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.217.2
--webserver_port=0
--tserver_master_addrs=127.16.217.62:42287
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.217.2
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986536687152-17252-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:20:46.890967 3234 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:20:46.892185 3234 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:20:46.905988 3240 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:20:46.906023 3242 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:20:46.906138 3239 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:20:46.907195 3234 server_base.cc:1034] running on GCE node
I20260620 20:20:46.907728 3234 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:20:46.908658 3234 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:20:46.909853 3234 hybrid_clock.cc:648] HybridClock initialized: now 1781986846909803 us; error 36 us; skew 500 ppm
I20260620 20:20:46.910245 3234 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:20:46 dist-test-slave-2hpg krb5kdc[2994](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986846, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.217.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:20:46.913309 3234 init.cc:377] Logged in from keytab as kudu/127.16.217.2@KRBTEST.COM (short username kudu)
I20260620 20:20:46.914676 3234 webserver.cc:460] Webserver started at http://127.16.217.2:42483/ using document root <none> and password file <none>
I20260620 20:20:46.915282 3234 fs_manager.cc:359] Metadata directory not provided
I20260620 20:20:46.915380 3234 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:20:46.915638 3234 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:20:46.918419 3234 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986536687152-17252-0/minicluster-data/ts-1/data/instance:
uuid: "acde98460e7842aba027d320e8c0981f"
format_stamp: "Formatted at 2026-06-20 20:20:46 on dist-test-slave-2hpg"
server_key: "c6217d025fd00863c07c72ef3dd14df7"
server_key_iv: "4351da87d7da1b5f6d77219b6a4cab5f"
server_key_version: "encryptionkey@0"
I20260620 20:20:46.919127 3234 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986536687152-17252-0/minicluster-data/ts-1/wal/instance:
uuid: "acde98460e7842aba027d320e8c0981f"
format_stamp: "Formatted at 2026-06-20 20:20:46 on dist-test-slave-2hpg"
server_key: "c6217d025fd00863c07c72ef3dd14df7"
server_key_iv: "4351da87d7da1b5f6d77219b6a4cab5f"
server_key_version: "encryptionkey@0"
I20260620 20:20:46.924049 3234 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 20:20:46.927490 3249 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:20:46.928890 3234 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 20:20:46.929051 3234 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986536687152-17252-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986536687152-17252-0/minicluster-data/ts-1/wal
uuid: "acde98460e7842aba027d320e8c0981f"
format_stamp: "Formatted at 2026-06-20 20:20:46 on dist-test-slave-2hpg"
server_key: "c6217d025fd00863c07c72ef3dd14df7"
server_key_iv: "4351da87d7da1b5f6d77219b6a4cab5f"
server_key_version: "encryptionkey@0"
I20260620 20:20:46.929319 3234 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986536687152-17252-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986536687152-17252-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986536687152-17252-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:20:46.962289 3234 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:20:46.966789 3234 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:20:46.967048 3234 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:20:46.967981 3234 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:20:46.969261 3234 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:20:46.969334 3234 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:20:46.969426 3234 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:20:46.969496 3234 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:20:47.002611 3234 rpc_server.cc:307] RPC server started. Bound to: 127.16.217.2:39869
I20260620 20:20:47.002688 3361 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.217.2:39869 every 8 connection(s)
I20260620 20:20:47.003928 3234 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986536687152-17252-0/minicluster-data/ts-1/data/info.pb
I20260620 20:20:47.012903 17252 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu as pid 3234
I20260620 20:20:47.013100 17252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986536687152-17252-0/minicluster-data/ts-1/wal/instance
I20260620 20:20:47.013523 17252 external_mini_cluster.cc:1428] Setting key ec0b572875fa2249ea5658c517fb67dd
Jun 20 20:20:47 dist-test-slave-2hpg krb5kdc[2994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986846, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.217.2@KRBTEST.COM for kudu/127.16.217.62@KRBTEST.COM
I20260620 20:20:47.025813 3230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:47.006770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.217.2:46701 (local address 127.16.217.62:42287)
0620 20:20:47.006950 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0620 20:20:47.006955 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:20:47.008165 (+ 1210us) server_negotiation.cc:407] Connection header received
0620 20:20:47.010042 (+ 1877us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:20:47.010047 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:20:47.010130 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:20:47.010219 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:20:47.011728 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:47.012423 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:47.014215 (+ 1792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:47.014689 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:47.014788 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:20:47.018708 (+ 3920us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:20:47.018716 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:20:47.018724 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:20:47.018770 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:20:47.022607 (+ 3837us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:20:47.023342 (+ 735us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:20:47.023350 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:20:47.023356 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:20:47.023524 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:20:47.023966 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:20:47.023973 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:20:47.023979 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:20:47.024358 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:20:47.024515 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0620 20:20:47.025339 (+ 824us) server_negotiation.cc:299] Negotiation successful
0620 20:20:47.025553 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 20:20:47.027318 3364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:47.007278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:42287 (local address 127.16.217.2:46701)
0620 20:20:47.007987 (+ 709us) negotiation.cc:107] Waiting for socket to connect
0620 20:20:47.008031 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 20:20:47.009719 (+ 1688us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:20:47.010431 (+ 712us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:20:47.010440 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:20:47.010984 (+ 544us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:20:47.011552 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:20:47.011582 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:20:47.012601 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:20:47.012605 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:20:47.014065 (+ 1460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:20:47.014082 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:20:47.014905 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:20:47.014912 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:20:47.015197 (+ 285us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:20:47.016032 (+ 835us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:20:47.016055 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:20:47.018475 (+ 2420us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:20:47.022759 (+ 4284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:20:47.022767 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:20:47.022780 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:20:47.023203 (+ 423us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:20:47.023660 (+ 457us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:20:47.023663 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:20:47.023666 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:20:47.023850 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:20:47.024477 (+ 627us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:20:47.024483 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:20:47.024892 (+ 409us) client_negotiation.cc:769] Sending connection context
0620 20:20:47.025212 (+ 320us) client_negotiation.cc:240] Negotiation successful
0620 20:20:47.025823 (+ 611us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":297,"threads_started":1}
I20260620 20:20:47.028968 3362 heartbeater.cc:347] Connected to a master server at 127.16.217.62:42287
I20260620 20:20:47.029314 3362 heartbeater.cc:464] Registering TS with master...
I20260620 20:20:47.030143 3362 heartbeater.cc:511] Master 127.16.217.62:42287 requested a full tablet report, sending...
I20260620 20:20:47.031689 3039 ts_manager.cc:194] Registered new tserver with Master: acde98460e7842aba027d320e8c0981f (127.16.217.2:39869)
I20260620 20:20:47.032578 3039 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.217.2@KRBTEST.COM'} at 127.16.217.2:46701
WARNING: no policy specified for kudu/127.16.217.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.217.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.217.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.3.keytab.
Entry for principal kudu/127.16.217.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.3.keytab.
I20260620 20:20:47.069859 17252 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
/tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986536687152-17252-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986536687152-17252-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986536687152-17252-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986536687152-17252-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.217.3:0
--local_ip_for_outbound_sockets=127.16.217.3
--webserver_interface=127.16.217.3
--webserver_port=0
--tserver_master_addrs=127.16.217.62:42287
--keytab_file=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.3.keytab
--principal=kudu/127.16.217.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.217.20:42511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986536687152-17252-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986536687152-17252-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986536687152-17252-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:20:47.394937 3367 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:20:47.395246 3367 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:20:47.395407 3367 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:20:47.403018 3367 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:20:47.403163 3367 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:20:47.403332 3367 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.217.3
I20260620 20:20:47.413028 3367 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.217.20:42511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986536687152-17252-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986536687152-17252-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.3.keytab
--principal=kudu/127.16.217.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.217.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986536687152-17252-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.217.3
--webserver_port=0
--tserver_master_addrs=127.16.217.62:42287
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.217.3
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986536687152-17252-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:20:47.414346 3367 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:20:47.415639 3367 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:20:47.428067 3373 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:20:47.428244 3372 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:20:47.428102 3375 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:20:47.431185 3367 server_base.cc:1034] running on GCE node
I20260620 20:20:47.431752 3367 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:20:47.432736 3367 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:20:47.433966 3367 hybrid_clock.cc:648] HybridClock initialized: now 1781986847433917 us; error 39 us; skew 500 ppm
I20260620 20:20:47.434255 3367 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:20:47 dist-test-slave-2hpg krb5kdc[2994](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986847, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.217.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:20:47.437311 3367 init.cc:377] Logged in from keytab as kudu/127.16.217.3@KRBTEST.COM (short username kudu)
I20260620 20:20:47.438822 3367 webserver.cc:460] Webserver started at http://127.16.217.3:42623/ using document root <none> and password file <none>
I20260620 20:20:47.439471 3367 fs_manager.cc:359] Metadata directory not provided
I20260620 20:20:47.439575 3367 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:20:47.439823 3367 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:20:47.442735 3367 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986536687152-17252-0/minicluster-data/ts-2/data/instance:
uuid: "df601a089d134d0a88c563780946c579"
format_stamp: "Formatted at 2026-06-20 20:20:47 on dist-test-slave-2hpg"
server_key: "6802a2e649e0ea2e9aa4d31a27b32912"
server_key_iv: "90e627888a9fae740505bab268c967e3"
server_key_version: "encryptionkey@0"
I20260620 20:20:47.443436 3367 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986536687152-17252-0/minicluster-data/ts-2/wal/instance:
uuid: "df601a089d134d0a88c563780946c579"
format_stamp: "Formatted at 2026-06-20 20:20:47 on dist-test-slave-2hpg"
server_key: "6802a2e649e0ea2e9aa4d31a27b32912"
server_key_iv: "90e627888a9fae740505bab268c967e3"
server_key_version: "encryptionkey@0"
I20260620 20:20:47.448300 3367 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 20:20:47.451705 3382 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:20:47.452950 3367 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 20:20:47.453125 3367 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986536687152-17252-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986536687152-17252-0/minicluster-data/ts-2/wal
uuid: "df601a089d134d0a88c563780946c579"
format_stamp: "Formatted at 2026-06-20 20:20:47 on dist-test-slave-2hpg"
server_key: "6802a2e649e0ea2e9aa4d31a27b32912"
server_key_iv: "90e627888a9fae740505bab268c967e3"
server_key_version: "encryptionkey@0"
I20260620 20:20:47.453312 3367 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986536687152-17252-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986536687152-17252-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986536687152-17252-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:20:47.471078 3367 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:20:47.475492 3367 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:20:47.475834 3367 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:20:47.476892 3367 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:20:47.478654 3367 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:20:47.478737 3367 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:20:47.478901 3367 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:20:47.478962 3367 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:20:47.514613 3367 rpc_server.cc:307] RPC server started. Bound to: 127.16.217.3:35951
I20260620 20:20:47.514714 3494 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.217.3:35951 every 8 connection(s)
I20260620 20:20:47.516103 3367 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986536687152-17252-0/minicluster-data/ts-2/data/info.pb
I20260620 20:20:47.519963 3228 heartbeater.cc:503] Master 127.16.217.62:42287 was elected leader, sending a full tablet report...
I20260620 20:20:47.525602 17252 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu as pid 3367
I20260620 20:20:47.525789 17252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986536687152-17252-0/minicluster-data/ts-2/wal/instance
I20260620 20:20:47.526248 17252 external_mini_cluster.cc:1428] Setting key 422888cc63cac004b08ef9300d990338
Jun 20 20:20:47 dist-test-slave-2hpg krb5kdc[2994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986847, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.217.3@KRBTEST.COM for kudu/127.16.217.62@KRBTEST.COM
I20260620 20:20:47.536762 3230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:47.518608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.217.3:35057 (local address 127.16.217.62:42287)
0620 20:20:47.518802 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0620 20:20:47.518807 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:20:47.520114 (+ 1307us) server_negotiation.cc:407] Connection header received
0620 20:20:47.522506 (+ 2392us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:20:47.522510 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:20:47.522572 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:20:47.522657 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:20:47.524160 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:47.524887 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:47.526734 (+ 1847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:47.527283 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:47.527440 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:20:47.531106 (+ 3666us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:20:47.531111 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:20:47.531115 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:20:47.531146 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:20:47.533650 (+ 2504us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:20:47.534461 (+ 811us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:20:47.534466 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:20:47.534468 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:20:47.534634 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:20:47.535062 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:20:47.535066 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:20:47.535074 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:20:47.535334 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:20:47.535497 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0620 20:20:47.536153 (+ 656us) server_negotiation.cc:299] Negotiation successful
0620 20:20:47.536381 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 20:20:47.537678 3497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:47.519044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:42287 (local address 127.16.217.3:35057)
0620 20:20:47.519878 (+ 834us) negotiation.cc:107] Waiting for socket to connect
0620 20:20:47.519943 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0620 20:20:47.522256 (+ 2313us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:20:47.522870 (+ 614us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:20:47.522878 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:20:47.523411 (+ 533us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:20:47.523993 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:20:47.524023 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:20:47.525061 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:20:47.525065 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:20:47.526588 (+ 1523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:20:47.526601 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:20:47.527514 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:20:47.527524 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:20:47.527842 (+ 318us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:20:47.528624 (+ 782us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:20:47.528653 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:20:47.530885 (+ 2232us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:20:47.533808 (+ 2923us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:20:47.533816 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:20:47.533833 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:20:47.534323 (+ 490us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:20:47.534749 (+ 426us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:20:47.534755 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:20:47.534758 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:20:47.534944 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:20:47.535464 (+ 520us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:20:47.535471 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:20:47.535804 (+ 333us) client_negotiation.cc:769] Sending connection context
0620 20:20:47.536070 (+ 266us) client_negotiation.cc:240] Negotiation successful
0620 20:20:47.536595 (+ 525us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":506,"thread_start_us":312,"threads_started":1}
I20260620 20:20:47.539086 3495 heartbeater.cc:347] Connected to a master server at 127.16.217.62:42287
I20260620 20:20:47.539417 3495 heartbeater.cc:464] Registering TS with master...
I20260620 20:20:47.540593 3495 heartbeater.cc:511] Master 127.16.217.62:42287 requested a full tablet report, sending...
I20260620 20:20:47.542316 3039 ts_manager.cc:194] Registered new tserver with Master: df601a089d134d0a88c563780946c579 (127.16.217.3:35951)
I20260620 20:20:47.543179 3039 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.217.3@KRBTEST.COM'} at 127.16.217.3:35057
I20260620 20:20:47.553799 17252 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 20:20:47.570959 3230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:47.558042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34716 (local address 127.16.217.62:42287)
0620 20:20:47.558211 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0620 20:20:47.558218 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:20:47.558612 (+ 394us) server_negotiation.cc:407] Connection header received
0620 20:20:47.558792 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:20:47.558796 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:20:47.558872 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:20:47.558967 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:20:47.560303 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:47.561022 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:47.562377 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:47.562786 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:47.562930 (+ 144us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:20:47.565567 (+ 2637us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:20:47.565579 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:20:47.565583 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:20:47.565613 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:20:47.568178 (+ 2565us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:20:47.568921 (+ 743us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:20:47.568926 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:20:47.568935 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:20:47.569071 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:20:47.569502 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:20:47.569512 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:20:47.569514 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:20:47.569806 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:20:47.570133 (+ 327us) server_negotiation.cc:1035] Waiting for connection context
0620 20:20:47.570492 (+ 359us) server_negotiation.cc:299] Negotiation successful
0620 20:20:47.570676 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 20:20:47.575068 3039 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34716:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 20:20:47.582029 3039 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 20:20:47.607159 3509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:47.594596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.2:39869 (local address 127.0.0.1:56256)
0620 20:20:47.596995 (+ 2399us) negotiation.cc:107] Waiting for socket to connect
0620 20:20:47.597050 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0620 20:20:47.597283 (+ 233us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:20:47.599186 (+ 1903us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:20:47.599194 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:20:47.599232 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:20:47.599620 (+ 388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:20:47.599638 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:20:47.601549 (+ 1911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:20:47.601555 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:20:47.604126 (+ 2571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:20:47.604147 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:20:47.606047 (+ 1900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:20:47.606234 (+ 187us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:20:47.606409 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:20:47.606496 (+ 87us) client_negotiation.cc:769] Sending connection context
0620 20:20:47.606570 (+ 74us) client_negotiation.cc:240] Negotiation successful
0620 20:20:47.606701 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2079,"mutex_wait_us":9,"thread_start_us":4016,"threads_started":1}
I20260620 20:20:47.607734 3507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:47.594599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.1:35609 (local address 127.0.0.1:49588)
0620 20:20:47.598884 (+ 4285us) negotiation.cc:107] Waiting for socket to connect
0620 20:20:47.598902 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 20:20:47.599022 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:20:47.599982 (+ 960us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:20:47.599989 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:20:47.600014 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:20:47.600521 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:20:47.600538 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:20:47.602531 (+ 1993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:20:47.602538 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:20:47.604553 (+ 2015us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:20:47.604564 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:20:47.605852 (+ 1288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:20:47.605859 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:20:47.606158 (+ 299us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:20:47.606249 (+ 91us) client_negotiation.cc:769] Sending connection context
0620 20:20:47.606387 (+ 138us) client_negotiation.cc:240] Negotiation successful
0620 20:20:47.606594 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4169,"mutex_wait_us":1361,"thread_start_us":532,"threads_started":1}
I20260620 20:20:47.608024 3510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:47.595123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49588 (local address 127.16.217.1:35609)
0620 20:20:47.598096 (+ 2973us) server_negotiation.cc:206] Beginning negotiation
0620 20:20:47.598110 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 20:20:47.599154 (+ 1044us) server_negotiation.cc:407] Connection header received
0620 20:20:47.599262 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:20:47.599272 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:20:47.599512 (+ 240us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:20:47.599712 (+ 200us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:20:47.600728 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:47.602353 (+ 1625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:47.604699 (+ 2346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:47.605641 (+ 942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:47.605795 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:20:47.606066 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0620 20:20:47.606847 (+ 781us) server_negotiation.cc:299] Negotiation successful
0620 20:20:47.607029 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2720,"thread_start_us":2549,"threads_started":1}
I20260620 20:20:47.608181 3508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:47.594631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56256 (local address 127.16.217.2:39869)
0620 20:20:47.598504 (+ 3873us) server_negotiation.cc:206] Beginning negotiation
0620 20:20:47.598513 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 20:20:47.598547 (+ 34us) server_negotiation.cc:407] Connection header received
0620 20:20:47.598676 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:20:47.598686 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:20:47.598956 (+ 270us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:20:47.599805 (+ 849us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:20:47.600029 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:47.601419 (+ 1390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:47.604301 (+ 2882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:47.605777 (+ 1476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:47.607414 (+ 1637us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:20:47.607607 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0620 20:20:47.607681 (+ 74us) server_negotiation.cc:299] Negotiation successful
0620 20:20:47.607851 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3727,"thread_start_us":3624,"threads_started":1}
I20260620 20:20:47.610140 3512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:47.600044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.3:35951 (local address 127.0.0.1:42538)
0620 20:20:47.601079 (+ 1035us) negotiation.cc:107] Waiting for socket to connect
0620 20:20:47.601101 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 20:20:47.601230 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:20:47.602117 (+ 887us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:20:47.602125 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:20:47.602151 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:20:47.602600 (+ 449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:20:47.602613 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:20:47.604010 (+ 1397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:20:47.604014 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:20:47.606213 (+ 2199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:20:47.606229 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:20:47.609398 (+ 3169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:20:47.609403 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:20:47.609526 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:20:47.609544 (+ 18us) client_negotiation.cc:769] Sending connection context
0620 20:20:47.609652 (+ 108us) client_negotiation.cc:240] Negotiation successful
0620 20:20:47.609767 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":496,"mutex_wait_us":2,"thread_start_us":351,"threads_started":1}
I20260620 20:20:47.610186 3511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:47.599343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42538 (local address 127.16.217.3:35951)
0620 20:20:47.600056 (+ 713us) server_negotiation.cc:206] Beginning negotiation
0620 20:20:47.600064 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 20:20:47.601214 (+ 1150us) server_negotiation.cc:407] Connection header received
0620 20:20:47.601406 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:20:47.601413 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:20:47.601630 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:20:47.601802 (+ 172us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:20:47.602814 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:47.603862 (+ 1048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:47.606386 (+ 2524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:47.607823 (+ 1437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:47.607917 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:20:47.608147 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0620 20:20:47.609691 (+ 1544us) server_negotiation.cc:299] Negotiation successful
0620 20:20:47.609838 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":560,"thread_start_us":415,"threads_started":1}
I20260620 20:20:47.611848 3163 tablet_service.cc:1467] Processing CreateTablet for tablet 577088fde89945f393c64d989594a65e (DEFAULT_TABLE table=test-table [id=b7b53ed852504933aa340c28e99f825a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:20:47.613533 3163 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 577088fde89945f393c64d989594a65e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:20:47.613034 3297 tablet_service.cc:1467] Processing CreateTablet for tablet 577088fde89945f393c64d989594a65e (DEFAULT_TABLE table=test-table [id=b7b53ed852504933aa340c28e99f825a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:20:47.613538 3430 tablet_service.cc:1467] Processing CreateTablet for tablet 577088fde89945f393c64d989594a65e (DEFAULT_TABLE table=test-table [id=b7b53ed852504933aa340c28e99f825a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:20:47.614391 3297 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 577088fde89945f393c64d989594a65e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:20:47.614868 3430 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 577088fde89945f393c64d989594a65e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:20:47.626314 3513 tablet_bootstrap.cc:492] T 577088fde89945f393c64d989594a65e P 8134f7d52e5a4fbfbe15fe2e0bce6dc1: Bootstrap starting.
I20260620 20:20:47.628882 3514 tablet_bootstrap.cc:492] T 577088fde89945f393c64d989594a65e P acde98460e7842aba027d320e8c0981f: Bootstrap starting.
I20260620 20:20:47.629815 3515 tablet_bootstrap.cc:492] T 577088fde89945f393c64d989594a65e P df601a089d134d0a88c563780946c579: Bootstrap starting.
I20260620 20:20:47.630448 3513 tablet_bootstrap.cc:654] T 577088fde89945f393c64d989594a65e P 8134f7d52e5a4fbfbe15fe2e0bce6dc1: Neither blocks nor log segments found. Creating new log.
I20260620 20:20:47.631925 3513 log.cc:824] T 577088fde89945f393c64d989594a65e P 8134f7d52e5a4fbfbe15fe2e0bce6dc1: Log is configured to *not* fsync() on all Append() calls
I20260620 20:20:47.632777 3515 tablet_bootstrap.cc:654] T 577088fde89945f393c64d989594a65e P df601a089d134d0a88c563780946c579: Neither blocks nor log segments found. Creating new log.
I20260620 20:20:47.632982 3514 tablet_bootstrap.cc:654] T 577088fde89945f393c64d989594a65e P acde98460e7842aba027d320e8c0981f: Neither blocks nor log segments found. Creating new log.
I20260620 20:20:47.634040 3515 log.cc:824] T 577088fde89945f393c64d989594a65e P df601a089d134d0a88c563780946c579: Log is configured to *not* fsync() on all Append() calls
I20260620 20:20:47.634279 3514 log.cc:824] T 577088fde89945f393c64d989594a65e P acde98460e7842aba027d320e8c0981f: Log is configured to *not* fsync() on all Append() calls
I20260620 20:20:47.635462 3513 tablet_bootstrap.cc:492] T 577088fde89945f393c64d989594a65e P 8134f7d52e5a4fbfbe15fe2e0bce6dc1: No bootstrap required, opened a new log
I20260620 20:20:47.635802 3513 ts_tablet_manager.cc:1397] T 577088fde89945f393c64d989594a65e P 8134f7d52e5a4fbfbe15fe2e0bce6dc1: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.002s
I20260620 20:20:47.637061 3514 tablet_bootstrap.cc:492] T 577088fde89945f393c64d989594a65e P acde98460e7842aba027d320e8c0981f: No bootstrap required, opened a new log
I20260620 20:20:47.637072 3515 tablet_bootstrap.cc:492] T 577088fde89945f393c64d989594a65e P df601a089d134d0a88c563780946c579: No bootstrap required, opened a new log
I20260620 20:20:47.637320 3515 ts_tablet_manager.cc:1397] T 577088fde89945f393c64d989594a65e P df601a089d134d0a88c563780946c579: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.003s
I20260620 20:20:47.637320 3514 ts_tablet_manager.cc:1397] T 577088fde89945f393c64d989594a65e P acde98460e7842aba027d320e8c0981f: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260620 20:20:47.643424 3513 raft_consensus.cc:348] T 577088fde89945f393c64d989594a65e P 8134f7d52e5a4fbfbe15fe2e0bce6dc1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acde98460e7842aba027d320e8c0981f" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 39869 } } peers { permanent_uuid: "8134f7d52e5a4fbfbe15fe2e0bce6dc1" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35609 } } peers { permanent_uuid: "df601a089d134d0a88c563780946c579" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 35951 } }
I20260620 20:20:47.643831 3513 raft_consensus.cc:374] T 577088fde89945f393c64d989594a65e P 8134f7d52e5a4fbfbe15fe2e0bce6dc1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:20:47.643961 3513 raft_consensus.cc:729] T 577088fde89945f393c64d989594a65e P 8134f7d52e5a4fbfbe15fe2e0bce6dc1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8134f7d52e5a4fbfbe15fe2e0bce6dc1, State: Initialized, Role: FOLLOWER
I20260620 20:20:47.644601 3513 consensus_queue.cc:260] T 577088fde89945f393c64d989594a65e P 8134f7d52e5a4fbfbe15fe2e0bce6dc1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acde98460e7842aba027d320e8c0981f" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 39869 } } peers { permanent_uuid: "8134f7d52e5a4fbfbe15fe2e0bce6dc1" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35609 } } peers { permanent_uuid: "df601a089d134d0a88c563780946c579" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 35951 } }
I20260620 20:20:47.646476 3513 ts_tablet_manager.cc:1428] T 577088fde89945f393c64d989594a65e P 8134f7d52e5a4fbfbe15fe2e0bce6dc1: Time spent starting tablet: real 0.010s user 0.003s sys 0.007s
I20260620 20:20:47.648025 3515 raft_consensus.cc:348] T 577088fde89945f393c64d989594a65e P df601a089d134d0a88c563780946c579 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acde98460e7842aba027d320e8c0981f" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 39869 } } peers { permanent_uuid: "8134f7d52e5a4fbfbe15fe2e0bce6dc1" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35609 } } peers { permanent_uuid: "df601a089d134d0a88c563780946c579" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 35951 } }
I20260620 20:20:47.648025 3514 raft_consensus.cc:348] T 577088fde89945f393c64d989594a65e P acde98460e7842aba027d320e8c0981f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acde98460e7842aba027d320e8c0981f" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 39869 } } peers { permanent_uuid: "8134f7d52e5a4fbfbe15fe2e0bce6dc1" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35609 } } peers { permanent_uuid: "df601a089d134d0a88c563780946c579" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 35951 } }
I20260620 20:20:47.648442 3514 raft_consensus.cc:374] T 577088fde89945f393c64d989594a65e P acde98460e7842aba027d320e8c0981f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:20:47.648473 3515 raft_consensus.cc:374] T 577088fde89945f393c64d989594a65e P df601a089d134d0a88c563780946c579 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:20:47.648562 3515 raft_consensus.cc:729] T 577088fde89945f393c64d989594a65e P df601a089d134d0a88c563780946c579 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df601a089d134d0a88c563780946c579, State: Initialized, Role: FOLLOWER
I20260620 20:20:47.648566 3514 raft_consensus.cc:729] T 577088fde89945f393c64d989594a65e P acde98460e7842aba027d320e8c0981f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acde98460e7842aba027d320e8c0981f, State: Initialized, Role: FOLLOWER
I20260620 20:20:47.649148 3515 consensus_queue.cc:260] T 577088fde89945f393c64d989594a65e P df601a089d134d0a88c563780946c579 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acde98460e7842aba027d320e8c0981f" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 39869 } } peers { permanent_uuid: "8134f7d52e5a4fbfbe15fe2e0bce6dc1" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35609 } } peers { permanent_uuid: "df601a089d134d0a88c563780946c579" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 35951 } }
I20260620 20:20:47.649142 3514 consensus_queue.cc:260] T 577088fde89945f393c64d989594a65e P acde98460e7842aba027d320e8c0981f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acde98460e7842aba027d320e8c0981f" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 39869 } } peers { permanent_uuid: "8134f7d52e5a4fbfbe15fe2e0bce6dc1" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35609 } } peers { permanent_uuid: "df601a089d134d0a88c563780946c579" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 35951 } }
I20260620 20:20:47.650269 3495 heartbeater.cc:503] Master 127.16.217.62:42287 was elected leader, sending a full tablet report...
I20260620 20:20:47.650718 3362 heartbeater.cc:503] Master 127.16.217.62:42287 was elected leader, sending a full tablet report...
I20260620 20:20:47.650914 3515 ts_tablet_manager.cc:1428] T 577088fde89945f393c64d989594a65e P df601a089d134d0a88c563780946c579: Time spent starting tablet: real 0.013s user 0.009s sys 0.005s
I20260620 20:20:47.652650 3514 ts_tablet_manager.cc:1428] T 577088fde89945f393c64d989594a65e P acde98460e7842aba027d320e8c0981f: Time spent starting tablet: real 0.015s user 0.015s sys 0.000s
W20260620 20:20:47.742200 3229 tablet.cc:2369] T 577088fde89945f393c64d989594a65e P 8134f7d52e5a4fbfbe15fe2e0bce6dc1: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 20:20:47.755972 3363 tablet.cc:2369] T 577088fde89945f393c64d989594a65e P acde98460e7842aba027d320e8c0981f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 20:20:47.767845 3496 tablet.cc:2369] T 577088fde89945f393c64d989594a65e P df601a089d134d0a88c563780946c579: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 20:20:48.009821 3519 raft_consensus.cc:482] T 577088fde89945f393c64d989594a65e P 8134f7d52e5a4fbfbe15fe2e0bce6dc1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 20:20:48.010183 3519 raft_consensus.cc:504] T 577088fde89945f393c64d989594a65e P 8134f7d52e5a4fbfbe15fe2e0bce6dc1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acde98460e7842aba027d320e8c0981f" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 39869 } } peers { permanent_uuid: "8134f7d52e5a4fbfbe15fe2e0bce6dc1" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35609 } } peers { permanent_uuid: "df601a089d134d0a88c563780946c579" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 35951 } }
I20260620 20:20:48.011878 3519 leader_election.cc:288] T 577088fde89945f393c64d989594a65e P 8134f7d52e5a4fbfbe15fe2e0bce6dc1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers acde98460e7842aba027d320e8c0981f (127.16.217.2:39869), df601a089d134d0a88c563780946c579 (127.16.217.3:35951)
I20260620 20:20:48.019131 3522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:48.012219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.2:39869 (local address 127.16.217.1:45953)
0620 20:20:48.012770 (+ 551us) negotiation.cc:107] Waiting for socket to connect
0620 20:20:48.012786 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 20:20:48.013018 (+ 232us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:20:48.013598 (+ 580us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:20:48.013602 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:20:48.013632 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:20:48.014058 (+ 426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:20:48.014070 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:20:48.015346 (+ 1276us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:20:48.015352 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:20:48.017021 (+ 1669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:20:48.017033 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:20:48.018357 (+ 1324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:20:48.018364 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:20:48.018531 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:20:48.018565 (+ 34us) client_negotiation.cc:769] Sending connection context
0620 20:20:48.018665 (+ 100us) client_negotiation.cc:240] Negotiation successful
0620 20:20:48.018803 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":435,"thread_start_us":423,"threads_started":1}
I20260620 20:20:48.019129 3508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:48.012440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.217.1:45953 (local address 127.16.217.2:39869)
0620 20:20:48.012634 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0620 20:20:48.012642 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 20:20:48.012881 (+ 239us) server_negotiation.cc:407] Connection header received
0620 20:20:48.013199 (+ 318us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:20:48.013208 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:20:48.013289 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:20:48.013364 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:20:48.014229 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:48.015218 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:48.017178 (+ 1960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:48.018165 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:48.018282 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:20:48.018332 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0620 20:20:48.018733 (+ 401us) server_negotiation.cc:299] Negotiation successful
0620 20:20:48.018845 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 20:20:48.019840 3511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:48.013342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.217.1:38937 (local address 127.16.217.3:35951)
0620 20:20:48.013566 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0620 20:20:48.013572 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:20:48.013833 (+ 261us) server_negotiation.cc:407] Connection header received
0620 20:20:48.014038 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:20:48.014044 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:20:48.014120 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:20:48.014209 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:20:48.014869 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:48.015804 (+ 935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:48.017645 (+ 1841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:48.018740 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:48.018871 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:20:48.018948 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 20:20:48.019472 (+ 524us) server_negotiation.cc:299] Negotiation successful
0620 20:20:48.019599 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260620 20:20:48.019893 3523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:48.013099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.3:35951 (local address 127.16.217.1:38937)
0620 20:20:48.013728 (+ 629us) negotiation.cc:107] Waiting for socket to connect
0620 20:20:48.013743 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 20:20:48.013857 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:20:48.014236 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:20:48.014240 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:20:48.014263 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:20:48.014726 (+ 463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:20:48.014739 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:20:48.015948 (+ 1209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:20:48.015952 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:20:48.017467 (+ 1515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:20:48.017481 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:20:48.019073 (+ 1592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:20:48.019079 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:20:48.019270 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:20:48.019326 (+ 56us) client_negotiation.cc:769] Sending connection context
0620 20:20:48.019429 (+ 103us) client_negotiation.cc:240] Negotiation successful
0620 20:20:48.019577 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":526,"thread_start_us":310,"threads_started":1}
I20260620 20:20:48.020351 3317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "577088fde89945f393c64d989594a65e" candidate_uuid: "8134f7d52e5a4fbfbe15fe2e0bce6dc1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acde98460e7842aba027d320e8c0981f" is_pre_election: true
I20260620 20:20:48.020875 3317 raft_consensus.cc:2454] T 577088fde89945f393c64d989594a65e P acde98460e7842aba027d320e8c0981f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8134f7d52e5a4fbfbe15fe2e0bce6dc1 in term 0.
I20260620 20:20:48.020871 3450 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "577088fde89945f393c64d989594a65e" candidate_uuid: "8134f7d52e5a4fbfbe15fe2e0bce6dc1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df601a089d134d0a88c563780946c579" is_pre_election: true
I20260620 20:20:48.021294 3450 raft_consensus.cc:2454] T 577088fde89945f393c64d989594a65e P df601a089d134d0a88c563780946c579 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8134f7d52e5a4fbfbe15fe2e0bce6dc1 in term 0.
I20260620 20:20:48.021584 3117 leader_election.cc:302] T 577088fde89945f393c64d989594a65e P 8134f7d52e5a4fbfbe15fe2e0bce6dc1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8134f7d52e5a4fbfbe15fe2e0bce6dc1, acde98460e7842aba027d320e8c0981f; no voters:
I20260620 20:20:48.021991 3519 raft_consensus.cc:2781] T 577088fde89945f393c64d989594a65e P 8134f7d52e5a4fbfbe15fe2e0bce6dc1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 20:20:48.022161 3519 raft_consensus.cc:482] T 577088fde89945f393c64d989594a65e P 8134f7d52e5a4fbfbe15fe2e0bce6dc1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 20:20:48.022289 3519 raft_consensus.cc:3037] T 577088fde89945f393c64d989594a65e P 8134f7d52e5a4fbfbe15fe2e0bce6dc1 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:20:48.024498 3519 raft_consensus.cc:504] T 577088fde89945f393c64d989594a65e P 8134f7d52e5a4fbfbe15fe2e0bce6dc1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acde98460e7842aba027d320e8c0981f" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 39869 } } peers { permanent_uuid: "8134f7d52e5a4fbfbe15fe2e0bce6dc1" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35609 } } peers { permanent_uuid: "df601a089d134d0a88c563780946c579" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 35951 } }
I20260620 20:20:48.025110 3519 leader_election.cc:288] T 577088fde89945f393c64d989594a65e P 8134f7d52e5a4fbfbe15fe2e0bce6dc1 [CANDIDATE]: Term 1 election: Requested vote from peers acde98460e7842aba027d320e8c0981f (127.16.217.2:39869), df601a089d134d0a88c563780946c579 (127.16.217.3:35951)
I20260620 20:20:48.025619 3317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "577088fde89945f393c64d989594a65e" candidate_uuid: "8134f7d52e5a4fbfbe15fe2e0bce6dc1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acde98460e7842aba027d320e8c0981f"
I20260620 20:20:48.025709 3450 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "577088fde89945f393c64d989594a65e" candidate_uuid: "8134f7d52e5a4fbfbe15fe2e0bce6dc1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df601a089d134d0a88c563780946c579"
I20260620 20:20:48.025854 3317 raft_consensus.cc:3037] T 577088fde89945f393c64d989594a65e P acde98460e7842aba027d320e8c0981f [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:20:48.025924 3450 raft_consensus.cc:3037] T 577088fde89945f393c64d989594a65e P df601a089d134d0a88c563780946c579 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:20:48.029426 3317 raft_consensus.cc:2454] T 577088fde89945f393c64d989594a65e P acde98460e7842aba027d320e8c0981f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8134f7d52e5a4fbfbe15fe2e0bce6dc1 in term 1.
I20260620 20:20:48.029429 3450 raft_consensus.cc:2454] T 577088fde89945f393c64d989594a65e P df601a089d134d0a88c563780946c579 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8134f7d52e5a4fbfbe15fe2e0bce6dc1 in term 1.
I20260620 20:20:48.029819 3117 leader_election.cc:302] T 577088fde89945f393c64d989594a65e P 8134f7d52e5a4fbfbe15fe2e0bce6dc1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8134f7d52e5a4fbfbe15fe2e0bce6dc1, df601a089d134d0a88c563780946c579; no voters:
I20260620 20:20:48.030174 3519 raft_consensus.cc:2781] T 577088fde89945f393c64d989594a65e P 8134f7d52e5a4fbfbe15fe2e0bce6dc1 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:20:48.030655 3519 raft_consensus.cc:686] T 577088fde89945f393c64d989594a65e P 8134f7d52e5a4fbfbe15fe2e0bce6dc1 [term 1 LEADER]: Becoming Leader. State: Replica: 8134f7d52e5a4fbfbe15fe2e0bce6dc1, State: Running, Role: LEADER
I20260620 20:20:48.031237 3519 consensus_queue.cc:237] T 577088fde89945f393c64d989594a65e P 8134f7d52e5a4fbfbe15fe2e0bce6dc1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acde98460e7842aba027d320e8c0981f" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 39869 } } peers { permanent_uuid: "8134f7d52e5a4fbfbe15fe2e0bce6dc1" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35609 } } peers { permanent_uuid: "df601a089d134d0a88c563780946c579" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 35951 } }
I20260620 20:20:48.036111 3038 catalog_manager.cc:5495] T 577088fde89945f393c64d989594a65e P 8134f7d52e5a4fbfbe15fe2e0bce6dc1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8134f7d52e5a4fbfbe15fe2e0bce6dc1 (127.16.217.1). New cstate: current_term: 1 leader_uuid: "8134f7d52e5a4fbfbe15fe2e0bce6dc1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acde98460e7842aba027d320e8c0981f" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 39869 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8134f7d52e5a4fbfbe15fe2e0bce6dc1" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35609 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df601a089d134d0a88c563780946c579" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 35951 } health_report { overall_health: UNKNOWN } } }
I20260620 20:20:48.062564 3510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:48.057439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49592 (local address 127.16.217.1:35609)
0620 20:20:48.057625 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0620 20:20:48.057630 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:20:48.057645 (+ 15us) server_negotiation.cc:407] Connection header received
0620 20:20:48.057754 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:20:48.057765 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:20:48.057856 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:20:48.058048 (+ 192us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 20:20:48.058386 (+ 338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:48.059095 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:48.060492 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:48.060887 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:48.060979 (+ 92us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:20:48.061462 (+ 483us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 20:20:48.061993 (+ 531us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 20:20:48.062121 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0620 20:20:48.062226 (+ 105us) server_negotiation.cc:299] Negotiation successful
0620 20:20:48.062289 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 20:20:48.070498 3317 raft_consensus.cc:1261] T 577088fde89945f393c64d989594a65e P acde98460e7842aba027d320e8c0981f [term 1 FOLLOWER]: Refusing update from remote peer 8134f7d52e5a4fbfbe15fe2e0bce6dc1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:20:48.070564 3450 raft_consensus.cc:1261] T 577088fde89945f393c64d989594a65e P df601a089d134d0a88c563780946c579 [term 1 FOLLOWER]: Refusing update from remote peer 8134f7d52e5a4fbfbe15fe2e0bce6dc1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:20:48.071419 3524 consensus_queue.cc:1035] T 577088fde89945f393c64d989594a65e P 8134f7d52e5a4fbfbe15fe2e0bce6dc1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df601a089d134d0a88c563780946c579" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 35951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:20:48.071794 3519 consensus_queue.cc:1035] T 577088fde89945f393c64d989594a65e P 8134f7d52e5a4fbfbe15fe2e0bce6dc1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "acde98460e7842aba027d320e8c0981f" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 39869 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:20:48.082473 3527 mvcc.cc:205] Tried to move back new op lower bound from 7299018129687007232 to 7299018129540313088. Current Snapshot: MvccSnapshot[applied={T|T < 7299018129687007232}]
I20260620 20:20:48.099308 3038 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34716:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 20:20:48.099630 3038 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34716:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 20:20:48.103344 3038 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d05bde36c6ec4fb4ad274504ad322e9e: Sending DeleteTablet for 3 replicas of tablet 577088fde89945f393c64d989594a65e
I20260620 20:20:48.104565 3163 tablet_service.cc:1514] Processing DeleteTablet for tablet 577088fde89945f393c64d989594a65e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:20:48 UTC) from {username='kudu', principal='kudu/127.16.217.62@KRBTEST.COM'} at 127.0.0.1:49588
I20260620 20:20:48.104673 3297 tablet_service.cc:1514] Processing DeleteTablet for tablet 577088fde89945f393c64d989594a65e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:20:48 UTC) from {username='kudu', principal='kudu/127.16.217.62@KRBTEST.COM'} at 127.0.0.1:56256
I20260620 20:20:48.104895 3430 tablet_service.cc:1514] Processing DeleteTablet for tablet 577088fde89945f393c64d989594a65e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:20:48 UTC) from {username='kudu', principal='kudu/127.16.217.62@KRBTEST.COM'} at 127.0.0.1:42538
I20260620 20:20:48.108920 17252 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu pid 3100
I20260620 20:20:48.117789 3536 tablet_replica.cc:330] T 577088fde89945f393c64d989594a65e P 8134f7d52e5a4fbfbe15fe2e0bce6dc1: stopping tablet replica
I20260620 20:20:48.118417 3537 tablet_replica.cc:330] T 577088fde89945f393c64d989594a65e P acde98460e7842aba027d320e8c0981f: stopping tablet replica
I20260620 20:20:48.119120 3537 raft_consensus.cc:2229] T 577088fde89945f393c64d989594a65e P acde98460e7842aba027d320e8c0981f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 20:20:48.119674 3536 raft_consensus.cc:2229] T 577088fde89945f393c64d989594a65e P 8134f7d52e5a4fbfbe15fe2e0bce6dc1 [term 1 LEADER]: Raft consensus shutting down.
I20260620 20:20:48.120297 3537 raft_consensus.cc:2258] T 577088fde89945f393c64d989594a65e P acde98460e7842aba027d320e8c0981f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:20:48.121062 3536 raft_consensus.cc:2258] T 577088fde89945f393c64d989594a65e P 8134f7d52e5a4fbfbe15fe2e0bce6dc1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:20:48.123891 3537 ts_tablet_manager.cc:1905] T 577088fde89945f393c64d989594a65e P acde98460e7842aba027d320e8c0981f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:20:48.126171 3536 ts_tablet_manager.cc:1905] T 577088fde89945f393c64d989594a65e P 8134f7d52e5a4fbfbe15fe2e0bce6dc1: Deleting tablet data with delete state TABLET_DATA_DELETED
Jun 20 20:20:48 dist-test-slave-2hpg krb5kdc[2994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986845, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.217.1@KRBTEST.COM
I20260620 20:20:48.126818 3538 tablet_replica.cc:330] T 577088fde89945f393c64d989594a65e P df601a089d134d0a88c563780946c579: stopping tablet replica
I20260620 20:20:48.127279 3538 raft_consensus.cc:2229] T 577088fde89945f393c64d989594a65e P df601a089d134d0a88c563780946c579 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 20:20:48.127844 3538 raft_consensus.cc:2258] T 577088fde89945f393c64d989594a65e P df601a089d134d0a88c563780946c579 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:20:48.130462 3538 ts_tablet_manager.cc:1905] T 577088fde89945f393c64d989594a65e P df601a089d134d0a88c563780946c579: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:20:48.137029 3510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:48.110309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49602 (local address 127.16.217.1:35609)
0620 20:20:48.110591 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0620 20:20:48.110606 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 20:20:48.113984 (+ 3378us) server_negotiation.cc:407] Connection header received
0620 20:20:48.114209 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:20:48.114219 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:20:48.114343 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:20:48.114468 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:20:48.117028 (+ 2560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:48.118472 (+ 1444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:48.120575 (+ 2103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:48.121418 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:48.121543 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:20:48.127798 (+ 6255us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:20:48.127811 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:20:48.127836 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:20:48.127883 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:20:48.132657 (+ 4774us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:20:48.133632 (+ 975us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:20:48.133645 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:20:48.133648 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:20:48.133838 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:20:48.134426 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:20:48.134432 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:20:48.134435 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:20:48.134904 (+ 469us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:20:48.135224 (+ 320us) server_negotiation.cc:1035] Waiting for connection context
0620 20:20:48.136380 (+ 1156us) server_negotiation.cc:299] Negotiation successful
0620 20:20:48.136615 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260620 20:20:48.137894 3223 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:20:48.138577 3538 ts_tablet_manager.cc:1918] T 577088fde89945f393c64d989594a65e P df601a089d134d0a88c563780946c579: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 20:20:48.138831 3538 log.cc:1196] T 577088fde89945f393c64d989594a65e P df601a089d134d0a88c563780946c579: Deleting WAL directory at /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986536687152-17252-0/minicluster-data/ts-2/wal/wals/577088fde89945f393c64d989594a65e
I20260620 20:20:48.139809 3538 ts_tablet_manager.cc:1939] T 577088fde89945f393c64d989594a65e P df601a089d134d0a88c563780946c579: Deleting consensus metadata
I20260620 20:20:48.142304 3537 ts_tablet_manager.cc:1918] T 577088fde89945f393c64d989594a65e P acde98460e7842aba027d320e8c0981f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 20:20:48.142550 3537 log.cc:1196] T 577088fde89945f393c64d989594a65e P acde98460e7842aba027d320e8c0981f: Deleting WAL directory at /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986536687152-17252-0/minicluster-data/ts-1/wal/wals/577088fde89945f393c64d989594a65e
I20260620 20:20:48.142700 3024 catalog_manager.cc:4841] TS df601a089d134d0a88c563780946c579 (127.16.217.3:35951): tablet 577088fde89945f393c64d989594a65e (table test-table [id=b7b53ed852504933aa340c28e99f825a]) successfully deleted
I20260620 20:20:48.143101 3537 ts_tablet_manager.cc:1939] T 577088fde89945f393c64d989594a65e P acde98460e7842aba027d320e8c0981f: Deleting consensus metadata
I20260620 20:20:48.144516 3024 catalog_manager.cc:4841] TS acde98460e7842aba027d320e8c0981f (127.16.217.2:39869): tablet 577088fde89945f393c64d989594a65e (table test-table [id=b7b53ed852504933aa340c28e99f825a]) successfully deleted
I20260620 20:20:48.236778 17252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu with pid 3100
W20260620 20:20:48.256412 3023 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.16.217.1:35609 (error 108)
I20260620 20:20:48.257249 17252 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu pid 3234
I20260620 20:20:48.258832 3542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:48.257971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.1:35609 (local address 127.0.0.1:49610)
0620 20:20:48.258435 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0620 20:20:48.258596 (+ 161us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.217.1:35609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":340,"threads_started":1}
W20260620 20:20:48.259060 3023 catalog_manager.cc:4584] TS 8134f7d52e5a4fbfbe15fe2e0bce6dc1 (127.16.217.1:35609): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 577088fde89945f393c64d989594a65e: Network error: Client connection negotiation failed: client connection to 127.16.217.1:35609: connect: Connection refused (error 111)
Jun 20 20:20:48 dist-test-slave-2hpg krb5kdc[2994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986845, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.217.2@KRBTEST.COM
I20260620 20:20:48.272866 3508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:48.258036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56268 (local address 127.16.217.2:39869)
0620 20:20:48.258266 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0620 20:20:48.258274 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 20:20:48.258297 (+ 23us) server_negotiation.cc:407] Connection header received
0620 20:20:48.258373 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:20:48.258379 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:20:48.258468 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:20:48.258604 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:20:48.259969 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:48.260789 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:48.262254 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:48.262694 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:48.262805 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:20:48.266915 (+ 4110us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:20:48.266928 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:20:48.266943 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:20:48.266985 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:20:48.270255 (+ 3270us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:20:48.270942 (+ 687us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:20:48.270950 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:20:48.270953 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:20:48.271112 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:20:48.271477 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:20:48.271481 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:20:48.271483 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:20:48.271872 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:20:48.272117 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0620 20:20:48.272432 (+ 315us) server_negotiation.cc:299] Negotiation successful
0620 20:20:48.272589 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260620 20:20:48.273542 3357 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:20:48.312826 3542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:48.312287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.1:35609 (local address 127.0.0.1:49622)
0620 20:20:48.312531 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0620 20:20:48.312658 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.217.1:35609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260620 20:20:48.313983 3542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:48.313527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.1:35609 (local address 127.0.0.1:49634)
0620 20:20:48.313686 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0620 20:20:48.313778 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.217.1:35609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260620 20:20:48.363595 17252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu with pid 3234
I20260620 20:20:48.374687 3542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:48.374065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.1:35609 (local address 127.0.0.1:49638)
0620 20:20:48.374351 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0620 20:20:48.374476 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.217.1:35609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260620 20:20:48.375746 3542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:48.375380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.1:35609 (local address 127.0.0.1:49642)
0620 20:20:48.375522 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0620 20:20:48.375621 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.217.1:35609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260620 20:20:48.381632 17252 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu pid 3367
Jun 20 20:20:48 dist-test-slave-2hpg krb5kdc[2994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986845, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.217.3@KRBTEST.COM
I20260620 20:20:48.397558 3511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:48.382390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42548 (local address 127.16.217.3:35951)
0620 20:20:48.382585 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0620 20:20:48.382592 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:20:48.382612 (+ 20us) server_negotiation.cc:407] Connection header received
0620 20:20:48.382824 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:20:48.382834 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:20:48.382907 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:20:48.383032 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:20:48.384454 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:48.385160 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:48.386487 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:48.386858 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:48.387000 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:20:48.391242 (+ 4242us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:20:48.391258 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:20:48.391274 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:20:48.391312 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:20:48.394140 (+ 2828us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:20:48.394753 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:20:48.394760 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:20:48.394762 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:20:48.394952 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:20:48.395412 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:20:48.395419 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:20:48.395424 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:20:48.395958 (+ 534us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:20:48.396284 (+ 326us) server_negotiation.cc:1035] Waiting for connection context
0620 20:20:48.396911 (+ 627us) server_negotiation.cc:299] Negotiation successful
0620 20:20:48.397148 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 20:20:48.399067 3490 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:20:48.456992 3542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:48.456535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.1:35609 (local address 127.0.0.1:49652)
0620 20:20:48.456756 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0620 20:20:48.456843 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.217.1:35609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260620 20:20:48.458114 3542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:48.457691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.1:35609 (local address 127.0.0.1:49654)
0620 20:20:48.457895 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0620 20:20:48.457977 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.217.1:35609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260620 20:20:48.526038 17252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu with pid 3367
I20260620 20:20:48.542776 17252 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu pid 3007
I20260620 20:20:48.543965 3069 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:20:48.654803 17252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu with pid 3007
2026-06-20T20:20:48Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (4606 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-06-20T20:20:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T20:20:48Z Disabled control of system clock
I20260620 20:20:48.733220 17252 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
/tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986536687152-17252-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986536687152-17252-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986536687152-17252-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986536687152-17252-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.217.62:34063
--webserver_interface=127.16.217.62
--webserver_port=0
--builtin_ntp_servers=127.16.217.20:37633
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.217.62:34063
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 20:20:49.067998 3549 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:20:49.068265 3549 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:20:49.068343 3549 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:20:49.075569 3549 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:20:49.075663 3549 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:20:49.075699 3549 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:20:49.075752 3549 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:20:49.075798 3549 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:20:49.084551 3549 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.217.20:37633
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986536687152-17252-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986536687152-17252-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.217.62:34063
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.217.62:34063
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986536687152-17252-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.217.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986536687152-17252-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:20:49.085853 3549 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:20:49.087570 3549 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:20:49.099217 3557 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:20:49.099296 3554 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:20:49.099296 3555 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:20:49.100291 3549 server_base.cc:1034] running on GCE node
I20260620 20:20:49.101099 3549 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:20:49.102530 3549 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:20:49.103729 3549 hybrid_clock.cc:648] HybridClock initialized: now 1781986849103669 us; error 58 us; skew 500 ppm
I20260620 20:20:49.104085 3549 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 20:20:49.106525 3549 webserver.cc:460] Webserver started at http://127.16.217.62:39459/ using document root <none> and password file <none>
I20260620 20:20:49.107193 3549 fs_manager.cc:359] Metadata directory not provided
I20260620 20:20:49.107308 3549 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:20:49.107776 3549 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:20:49.112249 3549 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986536687152-17252-0/minicluster-data/master-0/data/instance:
uuid: "301775c4958246bcb58f1e9a9a4546c0"
format_stamp: "Formatted at 2026-06-20 20:20:49 on dist-test-slave-2hpg"
server_key: "fd91a69c5db4caa8e335a685a1d3cdb9"
server_key_iv: "3790a6706ca011d4d15473d501cca0bf"
server_key_version: "encryptionkey@0"
I20260620 20:20:49.113433 3549 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986536687152-17252-0/minicluster-data/master-0/wal/instance:
uuid: "301775c4958246bcb58f1e9a9a4546c0"
format_stamp: "Formatted at 2026-06-20 20:20:49 on dist-test-slave-2hpg"
server_key: "fd91a69c5db4caa8e335a685a1d3cdb9"
server_key_iv: "3790a6706ca011d4d15473d501cca0bf"
server_key_version: "encryptionkey@0"
I20260620 20:20:49.119271 3549 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260620 20:20:49.122788 3563 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:20:49.124155 3549 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260620 20:20:49.124331 3549 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986536687152-17252-0/minicluster-data/master-0/data,/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986536687152-17252-0/minicluster-data/master-0/wal
uuid: "301775c4958246bcb58f1e9a9a4546c0"
format_stamp: "Formatted at 2026-06-20 20:20:49 on dist-test-slave-2hpg"
server_key: "fd91a69c5db4caa8e335a685a1d3cdb9"
server_key_iv: "3790a6706ca011d4d15473d501cca0bf"
server_key_version: "encryptionkey@0"
I20260620 20:20:49.124548 3549 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986536687152-17252-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986536687152-17252-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986536687152-17252-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:20:49.140345 3549 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:20:49.141182 3549 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:20:49.141482 3549 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:20:49.160391 3549 rpc_server.cc:307] RPC server started. Bound to: 127.16.217.62:34063
I20260620 20:20:49.160411 3614 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.217.62:34063 every 8 connection(s)
I20260620 20:20:49.161804 3549 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986536687152-17252-0/minicluster-data/master-0/data/info.pb
I20260620 20:20:49.165542 3615 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:20:49.166950 17252 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu as pid 3549
I20260620 20:20:49.167135 17252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986536687152-17252-0/minicluster-data/master-0/wal/instance
I20260620 20:20:49.167719 17252 external_mini_cluster.cc:1428] Setting key d7bb8cb6779ee082c91f8caf8bf9e793
I20260620 20:20:49.179865 3615 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 301775c4958246bcb58f1e9a9a4546c0: Bootstrap starting.
I20260620 20:20:49.181453 3618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:49.169633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40160 (local address 127.16.217.62:34063)
0620 20:20:49.170761 (+ 1128us) server_negotiation.cc:206] Beginning negotiation
0620 20:20:49.170772 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 20:20:49.170818 (+ 46us) server_negotiation.cc:407] Connection header received
0620 20:20:49.172385 (+ 1567us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:20:49.172406 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:20:49.172873 (+ 467us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:20:49.173201 (+ 328us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:20:49.173796 (+ 595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:49.175129 (+ 1333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:49.177074 (+ 1945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:49.177781 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:49.177984 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:20:49.179159 (+ 1175us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:20:49.179166 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:20:49.179179 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:20:49.179768 (+ 589us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:20:49.179844 (+ 76us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:20:49.179868 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:20:49.179951 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0620 20:20:49.180206 (+ 255us) server_negotiation.cc:299] Negotiation successful
0620 20:20:49.180534 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":681,"thread_start_us":431,"threads_started":1}
I20260620 20:20:49.183205 3615 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 301775c4958246bcb58f1e9a9a4546c0: Neither blocks nor log segments found. Creating new log.
I20260620 20:20:49.184361 3615 log.cc:824] T 00000000000000000000000000000000 P 301775c4958246bcb58f1e9a9a4546c0: Log is configured to *not* fsync() on all Append() calls
I20260620 20:20:49.187260 3615 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 301775c4958246bcb58f1e9a9a4546c0: No bootstrap required, opened a new log
I20260620 20:20:49.197458 3615 raft_consensus.cc:348] T 00000000000000000000000000000000 P 301775c4958246bcb58f1e9a9a4546c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "301775c4958246bcb58f1e9a9a4546c0" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 34063 } }
I20260620 20:20:49.197840 3615 raft_consensus.cc:374] T 00000000000000000000000000000000 P 301775c4958246bcb58f1e9a9a4546c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:20:49.197970 3615 raft_consensus.cc:729] T 00000000000000000000000000000000 P 301775c4958246bcb58f1e9a9a4546c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 301775c4958246bcb58f1e9a9a4546c0, State: Initialized, Role: FOLLOWER
I20260620 20:20:49.198523 3615 consensus_queue.cc:260] T 00000000000000000000000000000000 P 301775c4958246bcb58f1e9a9a4546c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "301775c4958246bcb58f1e9a9a4546c0" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 34063 } }
I20260620 20:20:49.198767 3615 raft_consensus.cc:388] T 00000000000000000000000000000000 P 301775c4958246bcb58f1e9a9a4546c0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 20:20:49.198907 3615 raft_consensus.cc:482] T 00000000000000000000000000000000 P 301775c4958246bcb58f1e9a9a4546c0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 20:20:49.199090 3615 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 301775c4958246bcb58f1e9a9a4546c0 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:20:49.201488 3615 raft_consensus.cc:504] T 00000000000000000000000000000000 P 301775c4958246bcb58f1e9a9a4546c0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "301775c4958246bcb58f1e9a9a4546c0" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 34063 } }
I20260620 20:20:49.202073 3615 leader_election.cc:302] T 00000000000000000000000000000000 P 301775c4958246bcb58f1e9a9a4546c0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 301775c4958246bcb58f1e9a9a4546c0; no voters:
I20260620 20:20:49.202670 3615 leader_election.cc:288] T 00000000000000000000000000000000 P 301775c4958246bcb58f1e9a9a4546c0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 20:20:49.202819 3620 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 301775c4958246bcb58f1e9a9a4546c0 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:20:49.203553 3620 raft_consensus.cc:686] T 00000000000000000000000000000000 P 301775c4958246bcb58f1e9a9a4546c0 [term 1 LEADER]: Becoming Leader. State: Replica: 301775c4958246bcb58f1e9a9a4546c0, State: Running, Role: LEADER
I20260620 20:20:49.204118 3620 consensus_queue.cc:237] T 00000000000000000000000000000000 P 301775c4958246bcb58f1e9a9a4546c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "301775c4958246bcb58f1e9a9a4546c0" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 34063 } }
I20260620 20:20:49.204788 3615 sys_catalog.cc:564] T 00000000000000000000000000000000 P 301775c4958246bcb58f1e9a9a4546c0 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:20:49.207078 3622 sys_catalog.cc:455] T 00000000000000000000000000000000 P 301775c4958246bcb58f1e9a9a4546c0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 301775c4958246bcb58f1e9a9a4546c0. Latest consensus state: current_term: 1 leader_uuid: "301775c4958246bcb58f1e9a9a4546c0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "301775c4958246bcb58f1e9a9a4546c0" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 34063 } } }
I20260620 20:20:49.207564 3622 sys_catalog.cc:458] T 00000000000000000000000000000000 P 301775c4958246bcb58f1e9a9a4546c0 [sys.catalog]: This master's current role is: LEADER
I20260620 20:20:49.208992 3621 sys_catalog.cc:455] T 00000000000000000000000000000000 P 301775c4958246bcb58f1e9a9a4546c0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "301775c4958246bcb58f1e9a9a4546c0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "301775c4958246bcb58f1e9a9a4546c0" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 34063 } } }
I20260620 20:20:49.209483 3621 sys_catalog.cc:458] T 00000000000000000000000000000000 P 301775c4958246bcb58f1e9a9a4546c0 [sys.catalog]: This master's current role is: LEADER
I20260620 20:20:49.210197 3629 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 20:20:49.215709 3629 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 20:20:49.224100 3629 catalog_manager.cc:1347] Generated new cluster ID: 67d9d523099c466596c4f86dbc719a5e
I20260620 20:20:49.224211 3629 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 20:20:49.241568 3629 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 20:20:49.242862 3629 catalog_manager.cc:1504] Loading token signing keys...
I20260620 20:20:49.252022 3629 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 301775c4958246bcb58f1e9a9a4546c0: Generated new TSK 0
I20260620 20:20:49.253088 3629 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 20:20:49.268095 17252 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
/tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986536687152-17252-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986536687152-17252-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986536687152-17252-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986536687152-17252-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.217.1:0
--local_ip_for_outbound_sockets=127.16.217.1
--webserver_interface=127.16.217.1
--webserver_port=0
--tserver_master_addrs=127.16.217.62:34063
--builtin_ntp_servers=127.16.217.20:37633
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 20:20:49.608685 3639 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:20:49.609012 3639 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:20:49.609171 3639 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:20:49.616946 3639 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:20:49.617092 3639 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:20:49.617268 3639 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.217.1
I20260620 20:20:49.626766 3639 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.217.20:37633
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986536687152-17252-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986536687152-17252-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.217.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986536687152-17252-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.217.1
--webserver_port=0
--tserver_master_addrs=127.16.217.62:34063
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.217.1
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986536687152-17252-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:20:49.628191 3639 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:20:49.630206 3639 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:20:49.643571 3644 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:20:49.643604 3647 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:20:49.643651 3645 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:20:49.647624 3639 server_base.cc:1034] running on GCE node
I20260620 20:20:49.648205 3639 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:20:49.649151 3639 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:20:49.650348 3639 hybrid_clock.cc:648] HybridClock initialized: now 1781986849650296 us; error 41 us; skew 500 ppm
I20260620 20:20:49.650638 3639 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 20:20:49.652757 3639 webserver.cc:460] Webserver started at http://127.16.217.1:45281/ using document root <none> and password file <none>
I20260620 20:20:49.653383 3639 fs_manager.cc:359] Metadata directory not provided
I20260620 20:20:49.653465 3639 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:20:49.653735 3639 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:20:49.656579 3639 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986536687152-17252-0/minicluster-data/ts-0/data/instance:
uuid: "4ac8c33a74774f709570c82c6fa95b97"
format_stamp: "Formatted at 2026-06-20 20:20:49 on dist-test-slave-2hpg"
server_key: "ce093235a4d67da3a891a06540a27f14"
server_key_iv: "652ec2cdafbc605fcc49b81cec8cfc47"
server_key_version: "encryptionkey@0"
I20260620 20:20:49.657335 3639 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986536687152-17252-0/minicluster-data/ts-0/wal/instance:
uuid: "4ac8c33a74774f709570c82c6fa95b97"
format_stamp: "Formatted at 2026-06-20 20:20:49 on dist-test-slave-2hpg"
server_key: "ce093235a4d67da3a891a06540a27f14"
server_key_iv: "652ec2cdafbc605fcc49b81cec8cfc47"
server_key_version: "encryptionkey@0"
I20260620 20:20:49.662364 3639 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260620 20:20:49.665691 3653 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:20:49.667161 3639 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 20:20:49.667456 3639 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986536687152-17252-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986536687152-17252-0/minicluster-data/ts-0/wal
uuid: "4ac8c33a74774f709570c82c6fa95b97"
format_stamp: "Formatted at 2026-06-20 20:20:49 on dist-test-slave-2hpg"
server_key: "ce093235a4d67da3a891a06540a27f14"
server_key_iv: "652ec2cdafbc605fcc49b81cec8cfc47"
server_key_version: "encryptionkey@0"
I20260620 20:20:49.667709 3639 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986536687152-17252-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986536687152-17252-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986536687152-17252-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:20:49.685623 3639 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:20:49.686471 3639 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:20:49.686767 3639 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:20:49.687696 3639 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:20:49.689131 3639 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:20:49.689252 3639 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:20:49.689368 3639 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:20:49.689450 3639 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:20:49.723029 3639 rpc_server.cc:307] RPC server started. Bound to: 127.16.217.1:43063
I20260620 20:20:49.723081 3765 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.217.1:43063 every 8 connection(s)
I20260620 20:20:49.724361 3639 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986536687152-17252-0/minicluster-data/ts-0/data/info.pb
I20260620 20:20:49.724730 17252 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu as pid 3639
I20260620 20:20:49.724903 17252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986536687152-17252-0/minicluster-data/ts-0/wal/instance
I20260620 20:20:49.725486 17252 external_mini_cluster.cc:1428] Setting key e423181f8efc578982bb8a4f6a88553e
I20260620 20:20:49.728648 17252 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
/tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986536687152-17252-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986536687152-17252-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986536687152-17252-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986536687152-17252-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.217.2:0
--local_ip_for_outbound_sockets=127.16.217.2
--webserver_interface=127.16.217.2
--webserver_port=0
--tserver_master_addrs=127.16.217.62:34063
--builtin_ntp_servers=127.16.217.20:37633
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 20:20:49.738647 3768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:49.727129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.217.1:35033 (local address 127.16.217.62:34063)
0620 20:20:49.727610 (+ 481us) server_negotiation.cc:206] Beginning negotiation
0620 20:20:49.727618 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 20:20:49.729105 (+ 1487us) server_negotiation.cc:407] Connection header received
0620 20:20:49.730934 (+ 1829us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:20:49.730942 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:20:49.731030 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:20:49.731158 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:20:49.732231 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:49.733434 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:49.735137 (+ 1703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:49.735849 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:49.735996 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:20:49.737621 (+ 1625us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:20:49.737630 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:20:49.737638 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:20:49.737663 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:20:49.737713 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:20:49.737732 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:20:49.737823 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0620 20:20:49.738076 (+ 253us) server_negotiation.cc:299] Negotiation successful
0620 20:20:49.738187 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":342,"thread_start_us":244,"threads_started":1}
I20260620 20:20:49.739410 3769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:49.727827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:34063 (local address 127.16.217.1:35033)
0620 20:20:49.728962 (+ 1135us) negotiation.cc:107] Waiting for socket to connect
0620 20:20:49.729005 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 20:20:49.730662 (+ 1657us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:20:49.731355 (+ 693us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:20:49.731363 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:20:49.731447 (+ 84us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:20:49.732064 (+ 617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:20:49.732080 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:20:49.733599 (+ 1519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:20:49.733603 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:20:49.734985 (+ 1382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:20:49.734998 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:20:49.736141 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:20:49.736150 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:20:49.736487 (+ 337us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:20:49.737238 (+ 751us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 20:20:49.737241 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:20:49.737304 (+ 63us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 20:20:49.737399 (+ 95us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:20:49.737831 (+ 432us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:20:49.737840 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:20:49.737846 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 20:20:49.738052 (+ 206us) client_negotiation.cc:240] Negotiation successful
0620 20:20:49.738337 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":531,"thread_start_us":352,"threads_started":1}
I20260620 20:20:49.741039 3766 heartbeater.cc:347] Connected to a master server at 127.16.217.62:34063
I20260620 20:20:49.741379 3766 heartbeater.cc:464] Registering TS with master...
I20260620 20:20:49.742237 3766 heartbeater.cc:511] Master 127.16.217.62:34063 requested a full tablet report, sending...
I20260620 20:20:49.744426 3580 ts_manager.cc:194] Registered new tserver with Master: 4ac8c33a74774f709570c82c6fa95b97 (127.16.217.1:43063)
I20260620 20:20:49.746098 3580 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.217.1:35033
W20260620 20:20:50.064972 3770 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:20:50.065266 3770 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:20:50.065362 3770 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:20:50.072747 3770 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:20:50.072854 3770 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:20:50.073061 3770 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.217.2
I20260620 20:20:50.082038 3770 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.217.20:37633
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986536687152-17252-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986536687152-17252-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.217.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986536687152-17252-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.217.2
--webserver_port=0
--tserver_master_addrs=127.16.217.62:34063
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.217.2
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986536687152-17252-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:20:50.083338 3770 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:20:50.085163 3770 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:20:50.098609 3776 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:20:50.098645 3775 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:20:50.099567 3778 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:20:50.100020 3770 server_base.cc:1034] running on GCE node
I20260620 20:20:50.100570 3770 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:20:50.101464 3770 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:20:50.102681 3770 hybrid_clock.cc:648] HybridClock initialized: now 1781986850102624 us; error 39 us; skew 500 ppm
I20260620 20:20:50.103073 3770 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 20:20:50.105579 3770 webserver.cc:460] Webserver started at http://127.16.217.2:40525/ using document root <none> and password file <none>
I20260620 20:20:50.106427 3770 fs_manager.cc:359] Metadata directory not provided
I20260620 20:20:50.106554 3770 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:20:50.106892 3770 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:20:50.109997 3770 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986536687152-17252-0/minicluster-data/ts-1/data/instance:
uuid: "a94f66671bd948a2999c243799a9c873"
format_stamp: "Formatted at 2026-06-20 20:20:50 on dist-test-slave-2hpg"
server_key: "a3de914e2ea55062203944445a4fd671"
server_key_iv: "c25936e2aeab56ac0eeea8d99791ba70"
server_key_version: "encryptionkey@0"
I20260620 20:20:50.110831 3770 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986536687152-17252-0/minicluster-data/ts-1/wal/instance:
uuid: "a94f66671bd948a2999c243799a9c873"
format_stamp: "Formatted at 2026-06-20 20:20:50 on dist-test-slave-2hpg"
server_key: "a3de914e2ea55062203944445a4fd671"
server_key_iv: "c25936e2aeab56ac0eeea8d99791ba70"
server_key_version: "encryptionkey@0"
I20260620 20:20:50.116056 3770 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260620 20:20:50.119505 3784 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:20:50.120761 3770 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 20:20:50.120937 3770 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986536687152-17252-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986536687152-17252-0/minicluster-data/ts-1/wal
uuid: "a94f66671bd948a2999c243799a9c873"
format_stamp: "Formatted at 2026-06-20 20:20:50 on dist-test-slave-2hpg"
server_key: "a3de914e2ea55062203944445a4fd671"
server_key_iv: "c25936e2aeab56ac0eeea8d99791ba70"
server_key_version: "encryptionkey@0"
I20260620 20:20:50.121150 3770 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986536687152-17252-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986536687152-17252-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986536687152-17252-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:20:50.156982 3770 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:20:50.158314 3770 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:20:50.158718 3770 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:20:50.160091 3770 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:20:50.161588 3770 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:20:50.161707 3770 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:20:50.161834 3770 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:20:50.161944 3770 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260620 20:20:50.205606 3770 rpc_server.cc:307] RPC server started. Bound to: 127.16.217.2:35183
I20260620 20:20:50.205610 3896 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.217.2:35183 every 8 connection(s)
I20260620 20:20:50.207736 3770 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986536687152-17252-0/minicluster-data/ts-1/data/info.pb
I20260620 20:20:50.216017 17252 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu as pid 3770
I20260620 20:20:50.216230 17252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986536687152-17252-0/minicluster-data/ts-1/wal/instance
I20260620 20:20:50.216859 17252 external_mini_cluster.cc:1428] Setting key 89f4bb64048f7a480a136e6e7065fc5b
I20260620 20:20:50.220461 17252 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
/tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986536687152-17252-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986536687152-17252-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986536687152-17252-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986536687152-17252-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.217.3:0
--local_ip_for_outbound_sockets=127.16.217.3
--webserver_interface=127.16.217.3
--webserver_port=0
--tserver_master_addrs=127.16.217.62:34063
--builtin_ntp_servers=127.16.217.20:37633
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 20:20:50.221768 3768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:50.210761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.217.2:36737 (local address 127.16.217.62:34063)
0620 20:20:50.210944 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0620 20:20:50.210948 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:20:50.212787 (+ 1839us) server_negotiation.cc:407] Connection header received
0620 20:20:50.214645 (+ 1858us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:20:50.214651 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:20:50.214741 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:20:50.214846 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:20:50.215846 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:50.216821 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:50.218918 (+ 2097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:50.219397 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:50.219473 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:20:50.220968 (+ 1495us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:20:50.220979 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:20:50.220982 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:20:50.220998 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:20:50.221030 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:20:50.221047 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:20:50.221136 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0620 20:20:50.221358 (+ 222us) server_negotiation.cc:299] Negotiation successful
0620 20:20:50.221454 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 20:20:50.222841 3899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:50.211655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:34063 (local address 127.16.217.2:36737)
0620 20:20:50.212630 (+ 975us) negotiation.cc:107] Waiting for socket to connect
0620 20:20:50.212666 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 20:20:50.214383 (+ 1717us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:20:50.215085 (+ 702us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:20:50.215092 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:20:50.215176 (+ 84us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:20:50.215691 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:20:50.215705 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:20:50.216986 (+ 1281us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:20:50.216991 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:20:50.218711 (+ 1720us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:20:50.218731 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:20:50.219700 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:20:50.219705 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:20:50.219901 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:20:50.220591 (+ 690us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 20:20:50.220597 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:20:50.220657 (+ 60us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 20:20:50.220752 (+ 95us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:20:50.221135 (+ 383us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:20:50.221147 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:20:50.221157 (+ 10us) client_negotiation.cc:769] Sending connection context
0620 20:20:50.221348 (+ 191us) client_negotiation.cc:240] Negotiation successful
0620 20:20:50.221718 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":546,"thread_start_us":336,"threads_started":1}
I20260620 20:20:50.224212 3897 heartbeater.cc:347] Connected to a master server at 127.16.217.62:34063
I20260620 20:20:50.224510 3897 heartbeater.cc:464] Registering TS with master...
I20260620 20:20:50.225435 3897 heartbeater.cc:511] Master 127.16.217.62:34063 requested a full tablet report, sending...
I20260620 20:20:50.226845 3580 ts_manager.cc:194] Registered new tserver with Master: a94f66671bd948a2999c243799a9c873 (127.16.217.2:35183)
I20260620 20:20:50.227682 3580 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.217.2:36737
W20260620 20:20:50.559463 3900 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:20:50.559755 3900 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:20:50.559846 3900 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:20:50.566977 3900 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:20:50.567080 3900 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:20:50.567236 3900 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.217.3
I20260620 20:20:50.576382 3900 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.217.20:37633
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986536687152-17252-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986536687152-17252-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.217.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986536687152-17252-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.217.3
--webserver_port=0
--tserver_master_addrs=127.16.217.62:34063
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.217.3
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986536687152-17252-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:20:50.577666 3900 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:20:50.579514 3900 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:20:50.592967 3905 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:20:50.593014 3906 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:20:50.592921 3908 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:20:50.594105 3900 server_base.cc:1034] running on GCE node
I20260620 20:20:50.594731 3900 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:20:50.595696 3900 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:20:50.596904 3900 hybrid_clock.cc:648] HybridClock initialized: now 1781986850596845 us; error 50 us; skew 500 ppm
I20260620 20:20:50.597244 3900 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 20:20:50.599668 3900 webserver.cc:460] Webserver started at http://127.16.217.3:35673/ using document root <none> and password file <none>
I20260620 20:20:50.600371 3900 fs_manager.cc:359] Metadata directory not provided
I20260620 20:20:50.600497 3900 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:20:50.600800 3900 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:20:50.603644 3900 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986536687152-17252-0/minicluster-data/ts-2/data/instance:
uuid: "8b72aba0078b42018e69e8ec17932cb7"
format_stamp: "Formatted at 2026-06-20 20:20:50 on dist-test-slave-2hpg"
server_key: "4cc3e1e791f4426c4402a29c64ff4b34"
server_key_iv: "d468f235754fefefb1e7378d77431a31"
server_key_version: "encryptionkey@0"
I20260620 20:20:50.604401 3900 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986536687152-17252-0/minicluster-data/ts-2/wal/instance:
uuid: "8b72aba0078b42018e69e8ec17932cb7"
format_stamp: "Formatted at 2026-06-20 20:20:50 on dist-test-slave-2hpg"
server_key: "4cc3e1e791f4426c4402a29c64ff4b34"
server_key_iv: "d468f235754fefefb1e7378d77431a31"
server_key_version: "encryptionkey@0"
I20260620 20:20:50.609601 3900 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260620 20:20:50.613189 3914 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:20:50.614522 3900 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 20:20:50.614683 3900 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986536687152-17252-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986536687152-17252-0/minicluster-data/ts-2/wal
uuid: "8b72aba0078b42018e69e8ec17932cb7"
format_stamp: "Formatted at 2026-06-20 20:20:50 on dist-test-slave-2hpg"
server_key: "4cc3e1e791f4426c4402a29c64ff4b34"
server_key_iv: "d468f235754fefefb1e7378d77431a31"
server_key_version: "encryptionkey@0"
I20260620 20:20:50.614879 3900 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986536687152-17252-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986536687152-17252-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986536687152-17252-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:20:50.631498 3900 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:20:50.632246 3900 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:20:50.632537 3900 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:20:50.633430 3900 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:20:50.634895 3900 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:20:50.635020 3900 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:20:50.635143 3900 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:20:50.635226 3900 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:20:50.667487 3900 rpc_server.cc:307] RPC server started. Bound to: 127.16.217.3:36525
I20260620 20:20:50.667599 4026 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.217.3:36525 every 8 connection(s)
I20260620 20:20:50.668860 3900 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986536687152-17252-0/minicluster-data/ts-2/data/info.pb
I20260620 20:20:50.679198 17252 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu as pid 3900
I20260620 20:20:50.679378 17252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986536687152-17252-0/minicluster-data/ts-2/wal/instance
I20260620 20:20:50.679949 17252 external_mini_cluster.cc:1428] Setting key 66e9cbcdbbde68466e2888b64ed5611e
I20260620 20:20:50.681077 3768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:50.671413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.217.3:49059 (local address 127.16.217.62:34063)
0620 20:20:50.671628 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0620 20:20:50.671634 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:20:50.673098 (+ 1464us) server_negotiation.cc:407] Connection header received
0620 20:20:50.674857 (+ 1759us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:20:50.674861 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:20:50.674928 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:20:50.675015 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:20:50.675992 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:50.676772 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:50.678311 (+ 1539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:50.678735 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:50.678861 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:20:50.680163 (+ 1302us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:20:50.680170 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:20:50.680175 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:20:50.680197 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:20:50.680234 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:20:50.680251 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:20:50.680343 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0620 20:20:50.680640 (+ 297us) server_negotiation.cc:299] Negotiation successful
0620 20:20:50.680747 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 20:20:50.682206 4029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:50.672043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:34063 (local address 127.16.217.3:49059)
0620 20:20:50.672949 (+ 906us) negotiation.cc:107] Waiting for socket to connect
0620 20:20:50.672988 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 20:20:50.674620 (+ 1632us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:20:50.675198 (+ 578us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:20:50.675205 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:20:50.675286 (+ 81us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:20:50.675860 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:20:50.675875 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:20:50.676907 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:20:50.676911 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:20:50.678193 (+ 1282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:20:50.678203 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:20:50.678925 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:20:50.678930 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:20:50.679112 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:20:50.679825 (+ 713us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 20:20:50.679827 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:20:50.679875 (+ 48us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 20:20:50.679949 (+ 74us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:20:50.680345 (+ 396us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:20:50.680370 (+ 25us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:20:50.680379 (+ 9us) client_negotiation.cc:769] Sending connection context
0620 20:20:50.680610 (+ 231us) client_negotiation.cc:240] Negotiation successful
0620 20:20:50.680942 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":494,"thread_start_us":305,"threads_started":1}
I20260620 20:20:50.683483 4027 heartbeater.cc:347] Connected to a master server at 127.16.217.62:34063
I20260620 20:20:50.683728 4027 heartbeater.cc:464] Registering TS with master...
I20260620 20:20:50.684351 4027 heartbeater.cc:511] Master 127.16.217.62:34063 requested a full tablet report, sending...
I20260620 20:20:50.685613 3580 ts_manager.cc:194] Registered new tserver with Master: 8b72aba0078b42018e69e8ec17932cb7 (127.16.217.3:36525)
I20260620 20:20:50.686411 3580 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.217.3:49059
I20260620 20:20:50.695104 17252 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 20:20:50.705382 3768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:50.699312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32772 (local address 127.16.217.62:34063)
0620 20:20:50.699548 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0620 20:20:50.699556 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 20:20:50.699965 (+ 409us) server_negotiation.cc:407] Connection header received
0620 20:20:50.700119 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:20:50.700123 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:20:50.700192 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:20:50.700272 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:20:50.700827 (+ 555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:50.701534 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:50.703108 (+ 1574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:50.703493 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:50.703625 (+ 132us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:20:50.704656 (+ 1031us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:20:50.704660 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:20:50.704664 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:20:50.704679 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:20:50.704733 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:20:50.704749 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:20:50.704862 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0620 20:20:50.705022 (+ 160us) server_negotiation.cc:299] Negotiation successful
0620 20:20:50.705086 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 20:20:50.709367 3580 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:32772:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 20:20:50.716605 3580 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 20:20:50.736951 4039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:50.729101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.1:43063 (local address 127.0.0.1:40412)
0620 20:20:50.729963 (+ 862us) negotiation.cc:107] Waiting for socket to connect
0620 20:20:50.730029 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0620 20:20:50.730236 (+ 207us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:20:50.731309 (+ 1073us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:20:50.731315 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:20:50.731344 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:20:50.731738 (+ 394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:20:50.731758 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:20:50.733043 (+ 1285us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:20:50.733051 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:20:50.734631 (+ 1580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:20:50.734644 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:20:50.735916 (+ 1272us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:20:50.735922 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:20:50.736125 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:20:50.736187 (+ 62us) client_negotiation.cc:769] Sending connection context
0620 20:20:50.736356 (+ 169us) client_negotiation.cc:240] Negotiation successful
0620 20:20:50.736594 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":514,"thread_start_us":319,"threads_started":1}
I20260620 20:20:50.738056 4040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:50.729357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40412 (local address 127.16.217.1:43063)
0620 20:20:50.730836 (+ 1479us) server_negotiation.cc:206] Beginning negotiation
0620 20:20:50.730842 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:20:50.730866 (+ 24us) server_negotiation.cc:407] Connection header received
0620 20:20:50.730937 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:20:50.730942 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:20:50.731115 (+ 173us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:20:50.731299 (+ 184us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:20:50.731917 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:50.732900 (+ 983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:50.734816 (+ 1916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:50.735724 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:50.735874 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:20:50.736128 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0620 20:20:50.737425 (+ 1297us) server_negotiation.cc:299] Negotiation successful
0620 20:20:50.737609 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1327,"thread_start_us":1222,"threads_started":1}
I20260620 20:20:50.742465 3701 tablet_service.cc:1467] Processing CreateTablet for tablet abc3a5c351914ae9b33ccd8216eb3cf9 (DEFAULT_TABLE table=test-table [id=098b720ecd994921ac19a2ac0a1e3120]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:20:50.743965 3701 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet abc3a5c351914ae9b33ccd8216eb3cf9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:20:50.744728 4042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:50.730464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54564 (local address 127.16.217.3:36525)
0620 20:20:50.737235 (+ 6771us) server_negotiation.cc:206] Beginning negotiation
0620 20:20:50.737243 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 20:20:50.737275 (+ 32us) server_negotiation.cc:407] Connection header received
0620 20:20:50.737367 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:20:50.737376 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:20:50.737610 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:20:50.738596 (+ 986us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:20:50.738891 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:50.740193 (+ 1302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:50.742813 (+ 2620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:50.743744 (+ 931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:50.743881 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:20:50.744160 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0620 20:20:50.744269 (+ 109us) server_negotiation.cc:299] Negotiation successful
0620 20:20:50.744413 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6603,"thread_start_us":6434,"threads_started":1}
I20260620 20:20:50.744730 4041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:50.730179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.3:36525 (local address 127.0.0.1:54564)
0620 20:20:50.736558 (+ 6379us) negotiation.cc:107] Waiting for socket to connect
0620 20:20:50.736581 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 20:20:50.736682 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:20:50.737833 (+ 1151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:20:50.737840 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:20:50.737908 (+ 68us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:20:50.738289 (+ 381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:20:50.738301 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:20:50.740347 (+ 2046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:20:50.740352 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:20:50.742615 (+ 2263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:20:50.742627 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:20:50.743931 (+ 1304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:20:50.743937 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:20:50.744094 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:20:50.744136 (+ 42us) client_negotiation.cc:769] Sending connection context
0620 20:20:50.744221 (+ 85us) client_negotiation.cc:240] Negotiation successful
0620 20:20:50.744344 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6240,"mutex_wait_us":64,"thread_start_us":6115,"threads_started":1}
I20260620 20:20:50.746165 4039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:50.737565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.2:35183 (local address 127.0.0.1:36738)
0620 20:20:50.737711 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0620 20:20:50.737726 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 20:20:50.737823 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:20:50.738560 (+ 737us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:20:50.738566 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:20:50.738592 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:20:50.738827 (+ 235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:20:50.738838 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:20:50.740331 (+ 1493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:20:50.740336 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:20:50.742612 (+ 2276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:20:50.742627 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:20:50.745359 (+ 2732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:20:50.745365 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:20:50.745536 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:20:50.745567 (+ 31us) client_negotiation.cc:769] Sending connection context
0620 20:20:50.745656 (+ 89us) client_negotiation.cc:240] Negotiation successful
0620 20:20:50.745793 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33,"mutex_wait_us":77}
I20260620 20:20:50.746222 4043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:50.737005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36738 (local address 127.16.217.2:35183)
0620 20:20:50.737521 (+ 516us) server_negotiation.cc:206] Beginning negotiation
0620 20:20:50.737531 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 20:20:50.737985 (+ 454us) server_negotiation.cc:407] Connection header received
0620 20:20:50.738095 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:20:50.738102 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:20:50.738323 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:20:50.738561 (+ 238us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:20:50.738976 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:50.740193 (+ 1217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:50.742784 (+ 2591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:50.745210 (+ 2426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:50.745347 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:20:50.745576 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0620 20:20:50.745691 (+ 115us) server_negotiation.cc:299] Negotiation successful
0620 20:20:50.745849 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":375,"thread_start_us":271,"threads_started":1}
I20260620 20:20:50.748711 3962 tablet_service.cc:1467] Processing CreateTablet for tablet abc3a5c351914ae9b33ccd8216eb3cf9 (DEFAULT_TABLE table=test-table [id=098b720ecd994921ac19a2ac0a1e3120]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:20:50.750033 3962 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet abc3a5c351914ae9b33ccd8216eb3cf9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:20:50.749936 3832 tablet_service.cc:1467] Processing CreateTablet for tablet abc3a5c351914ae9b33ccd8216eb3cf9 (DEFAULT_TABLE table=test-table [id=098b720ecd994921ac19a2ac0a1e3120]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:20:50.751230 3832 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet abc3a5c351914ae9b33ccd8216eb3cf9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:20:50.754145 3766 heartbeater.cc:503] Master 127.16.217.62:34063 was elected leader, sending a full tablet report...
I20260620 20:20:50.758419 4044 tablet_bootstrap.cc:492] T abc3a5c351914ae9b33ccd8216eb3cf9 P 4ac8c33a74774f709570c82c6fa95b97: Bootstrap starting.
I20260620 20:20:50.761710 4044 tablet_bootstrap.cc:654] T abc3a5c351914ae9b33ccd8216eb3cf9 P 4ac8c33a74774f709570c82c6fa95b97: Neither blocks nor log segments found. Creating new log.
I20260620 20:20:50.762825 4045 tablet_bootstrap.cc:492] T abc3a5c351914ae9b33ccd8216eb3cf9 P 8b72aba0078b42018e69e8ec17932cb7: Bootstrap starting.
I20260620 20:20:50.763140 4044 log.cc:824] T abc3a5c351914ae9b33ccd8216eb3cf9 P 4ac8c33a74774f709570c82c6fa95b97: Log is configured to *not* fsync() on all Append() calls
I20260620 20:20:50.766446 4045 tablet_bootstrap.cc:654] T abc3a5c351914ae9b33ccd8216eb3cf9 P 8b72aba0078b42018e69e8ec17932cb7: Neither blocks nor log segments found. Creating new log.
I20260620 20:20:50.766510 4044 tablet_bootstrap.cc:492] T abc3a5c351914ae9b33ccd8216eb3cf9 P 4ac8c33a74774f709570c82c6fa95b97: No bootstrap required, opened a new log
I20260620 20:20:50.766772 4044 ts_tablet_manager.cc:1397] T abc3a5c351914ae9b33ccd8216eb3cf9 P 4ac8c33a74774f709570c82c6fa95b97: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.003s
I20260620 20:20:50.767788 4045 log.cc:824] T abc3a5c351914ae9b33ccd8216eb3cf9 P 8b72aba0078b42018e69e8ec17932cb7: Log is configured to *not* fsync() on all Append() calls
I20260620 20:20:50.769781 4047 tablet_bootstrap.cc:492] T abc3a5c351914ae9b33ccd8216eb3cf9 P a94f66671bd948a2999c243799a9c873: Bootstrap starting.
I20260620 20:20:50.770831 4045 tablet_bootstrap.cc:492] T abc3a5c351914ae9b33ccd8216eb3cf9 P 8b72aba0078b42018e69e8ec17932cb7: No bootstrap required, opened a new log
I20260620 20:20:50.771111 4045 ts_tablet_manager.cc:1397] T abc3a5c351914ae9b33ccd8216eb3cf9 P 8b72aba0078b42018e69e8ec17932cb7: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.001s
I20260620 20:20:50.772837 4047 tablet_bootstrap.cc:654] T abc3a5c351914ae9b33ccd8216eb3cf9 P a94f66671bd948a2999c243799a9c873: Neither blocks nor log segments found. Creating new log.
I20260620 20:20:50.774099 4047 log.cc:824] T abc3a5c351914ae9b33ccd8216eb3cf9 P a94f66671bd948a2999c243799a9c873: Log is configured to *not* fsync() on all Append() calls
I20260620 20:20:50.776456 4047 tablet_bootstrap.cc:492] T abc3a5c351914ae9b33ccd8216eb3cf9 P a94f66671bd948a2999c243799a9c873: No bootstrap required, opened a new log
I20260620 20:20:50.776788 4047 ts_tablet_manager.cc:1397] T abc3a5c351914ae9b33ccd8216eb3cf9 P a94f66671bd948a2999c243799a9c873: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260620 20:20:50.777382 4044 raft_consensus.cc:348] T abc3a5c351914ae9b33ccd8216eb3cf9 P 4ac8c33a74774f709570c82c6fa95b97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ac8c33a74774f709570c82c6fa95b97" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 43063 } } peers { permanent_uuid: "8b72aba0078b42018e69e8ec17932cb7" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 36525 } } peers { permanent_uuid: "a94f66671bd948a2999c243799a9c873" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35183 } }
I20260620 20:20:50.777810 4044 raft_consensus.cc:374] T abc3a5c351914ae9b33ccd8216eb3cf9 P 4ac8c33a74774f709570c82c6fa95b97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:20:50.777969 4044 raft_consensus.cc:729] T abc3a5c351914ae9b33ccd8216eb3cf9 P 4ac8c33a74774f709570c82c6fa95b97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ac8c33a74774f709570c82c6fa95b97, State: Initialized, Role: FOLLOWER
I20260620 20:20:50.778664 4044 consensus_queue.cc:260] T abc3a5c351914ae9b33ccd8216eb3cf9 P 4ac8c33a74774f709570c82c6fa95b97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ac8c33a74774f709570c82c6fa95b97" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 43063 } } peers { permanent_uuid: "8b72aba0078b42018e69e8ec17932cb7" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 36525 } } peers { permanent_uuid: "a94f66671bd948a2999c243799a9c873" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35183 } }
I20260620 20:20:50.780382 4044 ts_tablet_manager.cc:1428] T abc3a5c351914ae9b33ccd8216eb3cf9 P 4ac8c33a74774f709570c82c6fa95b97: Time spent starting tablet: real 0.013s user 0.013s sys 0.000s
I20260620 20:20:50.781492 4045 raft_consensus.cc:348] T abc3a5c351914ae9b33ccd8216eb3cf9 P 8b72aba0078b42018e69e8ec17932cb7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ac8c33a74774f709570c82c6fa95b97" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 43063 } } peers { permanent_uuid: "8b72aba0078b42018e69e8ec17932cb7" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 36525 } } peers { permanent_uuid: "a94f66671bd948a2999c243799a9c873" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35183 } }
I20260620 20:20:50.781978 4045 raft_consensus.cc:374] T abc3a5c351914ae9b33ccd8216eb3cf9 P 8b72aba0078b42018e69e8ec17932cb7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:20:50.782100 4045 raft_consensus.cc:729] T abc3a5c351914ae9b33ccd8216eb3cf9 P 8b72aba0078b42018e69e8ec17932cb7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b72aba0078b42018e69e8ec17932cb7, State: Initialized, Role: FOLLOWER
I20260620 20:20:50.782778 4045 consensus_queue.cc:260] T abc3a5c351914ae9b33ccd8216eb3cf9 P 8b72aba0078b42018e69e8ec17932cb7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ac8c33a74774f709570c82c6fa95b97" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 43063 } } peers { permanent_uuid: "8b72aba0078b42018e69e8ec17932cb7" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 36525 } } peers { permanent_uuid: "a94f66671bd948a2999c243799a9c873" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35183 } }
I20260620 20:20:50.783814 4027 heartbeater.cc:503] Master 127.16.217.62:34063 was elected leader, sending a full tablet report...
I20260620 20:20:50.784461 4045 ts_tablet_manager.cc:1428] T abc3a5c351914ae9b33ccd8216eb3cf9 P 8b72aba0078b42018e69e8ec17932cb7: Time spent starting tablet: real 0.013s user 0.009s sys 0.004s
I20260620 20:20:50.786239 4047 raft_consensus.cc:348] T abc3a5c351914ae9b33ccd8216eb3cf9 P a94f66671bd948a2999c243799a9c873 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ac8c33a74774f709570c82c6fa95b97" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 43063 } } peers { permanent_uuid: "8b72aba0078b42018e69e8ec17932cb7" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 36525 } } peers { permanent_uuid: "a94f66671bd948a2999c243799a9c873" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35183 } }
I20260620 20:20:50.786687 4047 raft_consensus.cc:374] T abc3a5c351914ae9b33ccd8216eb3cf9 P a94f66671bd948a2999c243799a9c873 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:20:50.786850 4047 raft_consensus.cc:729] T abc3a5c351914ae9b33ccd8216eb3cf9 P a94f66671bd948a2999c243799a9c873 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a94f66671bd948a2999c243799a9c873, State: Initialized, Role: FOLLOWER
I20260620 20:20:50.787436 4047 consensus_queue.cc:260] T abc3a5c351914ae9b33ccd8216eb3cf9 P a94f66671bd948a2999c243799a9c873 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ac8c33a74774f709570c82c6fa95b97" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 43063 } } peers { permanent_uuid: "8b72aba0078b42018e69e8ec17932cb7" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 36525 } } peers { permanent_uuid: "a94f66671bd948a2999c243799a9c873" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35183 } }
I20260620 20:20:50.788489 3897 heartbeater.cc:503] Master 127.16.217.62:34063 was elected leader, sending a full tablet report...
I20260620 20:20:50.788899 4047 ts_tablet_manager.cc:1428] T abc3a5c351914ae9b33ccd8216eb3cf9 P a94f66671bd948a2999c243799a9c873: Time spent starting tablet: real 0.012s user 0.013s sys 0.000s
I20260620 20:20:50.908123 4052 raft_consensus.cc:482] T abc3a5c351914ae9b33ccd8216eb3cf9 P a94f66671bd948a2999c243799a9c873 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 20:20:50.908428 4052 raft_consensus.cc:504] T abc3a5c351914ae9b33ccd8216eb3cf9 P a94f66671bd948a2999c243799a9c873 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ac8c33a74774f709570c82c6fa95b97" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 43063 } } peers { permanent_uuid: "8b72aba0078b42018e69e8ec17932cb7" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 36525 } } peers { permanent_uuid: "a94f66671bd948a2999c243799a9c873" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35183 } }
I20260620 20:20:50.910331 4052 leader_election.cc:288] T abc3a5c351914ae9b33ccd8216eb3cf9 P a94f66671bd948a2999c243799a9c873 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4ac8c33a74774f709570c82c6fa95b97 (127.16.217.1:43063), 8b72aba0078b42018e69e8ec17932cb7 (127.16.217.3:36525)
I20260620 20:20:50.917991 4040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:50.910837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.217.2:59157 (local address 127.16.217.1:43063)
0620 20:20:50.911109 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0620 20:20:50.911115 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:20:50.911642 (+ 527us) server_negotiation.cc:407] Connection header received
0620 20:20:50.911951 (+ 309us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:20:50.911957 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:20:50.912069 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:20:50.912148 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:20:50.913062 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:50.914212 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:50.916279 (+ 2067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:50.917038 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:50.917215 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:20:50.917294 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 20:20:50.917586 (+ 292us) server_negotiation.cc:299] Negotiation successful
0620 20:20:50.917710 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119}
I20260620 20:20:50.918081 4053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:50.910579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.1:43063 (local address 127.16.217.2:59157)
0620 20:20:50.911489 (+ 910us) negotiation.cc:107] Waiting for socket to connect
0620 20:20:50.911551 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0620 20:20:50.911772 (+ 221us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:20:50.912355 (+ 583us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:20:50.912359 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:20:50.912391 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:20:50.912871 (+ 480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:20:50.912887 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:20:50.914390 (+ 1503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:20:50.914395 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:20:50.916082 (+ 1687us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:20:50.916093 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:20:50.917229 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:20:50.917235 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:20:50.917412 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:20:50.917436 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 20:20:50.917547 (+ 111us) client_negotiation.cc:240] Negotiation successful
0620 20:20:50.917690 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":740,"mutex_wait_us":3,"thread_start_us":409,"threads_started":1}
I20260620 20:20:50.918397 4054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:50.911501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.3:36525 (local address 127.16.217.2:58521)
0620 20:20:50.912616 (+ 1115us) negotiation.cc:107] Waiting for socket to connect
0620 20:20:50.912637 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 20:20:50.912770 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:20:50.913256 (+ 486us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:20:50.913261 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:20:50.913300 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:20:50.913686 (+ 386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:20:50.913698 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:20:50.915144 (+ 1446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:20:50.915147 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:20:50.916532 (+ 1385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:20:50.916543 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:20:50.917658 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:20:50.917664 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:20:50.917830 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:20:50.917852 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 20:20:50.917972 (+ 120us) client_negotiation.cc:240] Negotiation successful
0620 20:20:50.918100 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":992,"thread_start_us":726,"threads_started":1}
I20260620 20:20:50.918370 4042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:50.911937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.217.2:58521 (local address 127.16.217.3:36525)
0620 20:20:50.912164 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0620 20:20:50.912171 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:20:50.912725 (+ 554us) server_negotiation.cc:407] Connection header received
0620 20:20:50.912915 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:20:50.912921 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:20:50.913001 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:20:50.913097 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:20:50.913944 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:50.914981 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:50.916719 (+ 1738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:50.917470 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:50.917648 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:20:50.917712 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0620 20:20:50.918007 (+ 295us) server_negotiation.cc:299] Negotiation successful
0620 20:20:50.918153 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 20:20:50.919216 3721 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abc3a5c351914ae9b33ccd8216eb3cf9" candidate_uuid: "a94f66671bd948a2999c243799a9c873" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ac8c33a74774f709570c82c6fa95b97" is_pre_election: true
I20260620 20:20:50.919310 3982 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abc3a5c351914ae9b33ccd8216eb3cf9" candidate_uuid: "a94f66671bd948a2999c243799a9c873" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b72aba0078b42018e69e8ec17932cb7" is_pre_election: true
I20260620 20:20:50.919804 3982 raft_consensus.cc:2454] T abc3a5c351914ae9b33ccd8216eb3cf9 P 8b72aba0078b42018e69e8ec17932cb7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a94f66671bd948a2999c243799a9c873 in term 0.
I20260620 20:20:50.919809 3721 raft_consensus.cc:2454] T abc3a5c351914ae9b33ccd8216eb3cf9 P 4ac8c33a74774f709570c82c6fa95b97 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a94f66671bd948a2999c243799a9c873 in term 0.
W20260620 20:20:50.920535 4028 tablet.cc:2369] T abc3a5c351914ae9b33ccd8216eb3cf9 P 8b72aba0078b42018e69e8ec17932cb7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 20:20:50.920459 3785 leader_election.cc:302] T abc3a5c351914ae9b33ccd8216eb3cf9 P a94f66671bd948a2999c243799a9c873 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4ac8c33a74774f709570c82c6fa95b97, a94f66671bd948a2999c243799a9c873; no voters:
I20260620 20:20:50.920867 4052 raft_consensus.cc:2781] T abc3a5c351914ae9b33ccd8216eb3cf9 P a94f66671bd948a2999c243799a9c873 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 20:20:50.921013 4052 raft_consensus.cc:482] T abc3a5c351914ae9b33ccd8216eb3cf9 P a94f66671bd948a2999c243799a9c873 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 20:20:50.921126 4052 raft_consensus.cc:3037] T abc3a5c351914ae9b33ccd8216eb3cf9 P a94f66671bd948a2999c243799a9c873 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:20:50.923383 4052 raft_consensus.cc:504] T abc3a5c351914ae9b33ccd8216eb3cf9 P a94f66671bd948a2999c243799a9c873 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ac8c33a74774f709570c82c6fa95b97" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 43063 } } peers { permanent_uuid: "8b72aba0078b42018e69e8ec17932cb7" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 36525 } } peers { permanent_uuid: "a94f66671bd948a2999c243799a9c873" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35183 } }
I20260620 20:20:50.923867 4052 leader_election.cc:288] T abc3a5c351914ae9b33ccd8216eb3cf9 P a94f66671bd948a2999c243799a9c873 [CANDIDATE]: Term 1 election: Requested vote from peers 4ac8c33a74774f709570c82c6fa95b97 (127.16.217.1:43063), 8b72aba0078b42018e69e8ec17932cb7 (127.16.217.3:36525)
I20260620 20:20:50.924319 3721 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abc3a5c351914ae9b33ccd8216eb3cf9" candidate_uuid: "a94f66671bd948a2999c243799a9c873" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ac8c33a74774f709570c82c6fa95b97"
I20260620 20:20:50.924342 3982 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abc3a5c351914ae9b33ccd8216eb3cf9" candidate_uuid: "a94f66671bd948a2999c243799a9c873" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b72aba0078b42018e69e8ec17932cb7"
I20260620 20:20:50.924510 3721 raft_consensus.cc:3037] T abc3a5c351914ae9b33ccd8216eb3cf9 P 4ac8c33a74774f709570c82c6fa95b97 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:20:50.924516 3982 raft_consensus.cc:3037] T abc3a5c351914ae9b33ccd8216eb3cf9 P 8b72aba0078b42018e69e8ec17932cb7 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:20:50.926811 3721 raft_consensus.cc:2454] T abc3a5c351914ae9b33ccd8216eb3cf9 P 4ac8c33a74774f709570c82c6fa95b97 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a94f66671bd948a2999c243799a9c873 in term 1.
I20260620 20:20:50.926815 3982 raft_consensus.cc:2454] T abc3a5c351914ae9b33ccd8216eb3cf9 P 8b72aba0078b42018e69e8ec17932cb7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a94f66671bd948a2999c243799a9c873 in term 1.
I20260620 20:20:50.927199 3785 leader_election.cc:302] T abc3a5c351914ae9b33ccd8216eb3cf9 P a94f66671bd948a2999c243799a9c873 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4ac8c33a74774f709570c82c6fa95b97, a94f66671bd948a2999c243799a9c873; no voters:
I20260620 20:20:50.927469 4052 raft_consensus.cc:2781] T abc3a5c351914ae9b33ccd8216eb3cf9 P a94f66671bd948a2999c243799a9c873 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:20:50.927881 4052 raft_consensus.cc:686] T abc3a5c351914ae9b33ccd8216eb3cf9 P a94f66671bd948a2999c243799a9c873 [term 1 LEADER]: Becoming Leader. State: Replica: a94f66671bd948a2999c243799a9c873, State: Running, Role: LEADER
I20260620 20:20:50.928365 4052 consensus_queue.cc:237] T abc3a5c351914ae9b33ccd8216eb3cf9 P a94f66671bd948a2999c243799a9c873 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ac8c33a74774f709570c82c6fa95b97" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 43063 } } peers { permanent_uuid: "8b72aba0078b42018e69e8ec17932cb7" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 36525 } } peers { permanent_uuid: "a94f66671bd948a2999c243799a9c873" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35183 } }
I20260620 20:20:50.933454 3580 catalog_manager.cc:5495] T abc3a5c351914ae9b33ccd8216eb3cf9 P a94f66671bd948a2999c243799a9c873 reported cstate change: term changed from 0 to 1, leader changed from <none> to a94f66671bd948a2999c243799a9c873 (127.16.217.2). New cstate: current_term: 1 leader_uuid: "a94f66671bd948a2999c243799a9c873" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ac8c33a74774f709570c82c6fa95b97" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 43063 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8b72aba0078b42018e69e8ec17932cb7" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 36525 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a94f66671bd948a2999c243799a9c873" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35183 } health_report { overall_health: HEALTHY } } }
W20260620 20:20:50.959949 3898 tablet.cc:2369] T abc3a5c351914ae9b33ccd8216eb3cf9 P a94f66671bd948a2999c243799a9c873: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 20:20:50.976768 3767 tablet.cc:2369] T abc3a5c351914ae9b33ccd8216eb3cf9 P 4ac8c33a74774f709570c82c6fa95b97: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 20:20:50.991905 4043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:50.986227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36752 (local address 127.16.217.2:35183)
0620 20:20:50.986431 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0620 20:20:50.986439 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 20:20:50.986459 (+ 20us) server_negotiation.cc:407] Connection header received
0620 20:20:50.986720 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:20:50.986737 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:20:50.986827 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:20:50.986971 (+ 144us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 20:20:50.987643 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:50.988345 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:50.989669 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:50.990119 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:50.990246 (+ 127us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:20:50.990801 (+ 555us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 20:20:50.991314 (+ 513us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 20:20:50.991492 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0620 20:20:50.991645 (+ 153us) server_negotiation.cc:299] Negotiation successful
0620 20:20:50.991685 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 20:20:50.999660 3982 raft_consensus.cc:1261] T abc3a5c351914ae9b33ccd8216eb3cf9 P 8b72aba0078b42018e69e8ec17932cb7 [term 1 FOLLOWER]: Refusing update from remote peer a94f66671bd948a2999c243799a9c873: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:20:50.999716 3721 raft_consensus.cc:1261] T abc3a5c351914ae9b33ccd8216eb3cf9 P 4ac8c33a74774f709570c82c6fa95b97 [term 1 FOLLOWER]: Refusing update from remote peer a94f66671bd948a2999c243799a9c873: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:20:51.000574 4055 consensus_queue.cc:1035] T abc3a5c351914ae9b33ccd8216eb3cf9 P a94f66671bd948a2999c243799a9c873 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b72aba0078b42018e69e8ec17932cb7" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 36525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:20:51.000896 4052 consensus_queue.cc:1035] T abc3a5c351914ae9b33ccd8216eb3cf9 P a94f66671bd948a2999c243799a9c873 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ac8c33a74774f709570c82c6fa95b97" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 43063 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:20:51.013206 4058 mvcc.cc:205] Tried to move back new op lower bound from 7299018141684989952 to 7299018141406351360. Current Snapshot: MvccSnapshot[applied={T|T < 7299018141684989952}]
I20260620 20:20:51.018361 4060 mvcc.cc:205] Tried to move back new op lower bound from 7299018141684989952 to 7299018141406351360. Current Snapshot: MvccSnapshot[applied={T|T < 7299018141684989952}]
I20260620 20:20:51.018841 4059 mvcc.cc:205] Tried to move back new op lower bound from 7299018141684989952 to 7299018141406351360. Current Snapshot: MvccSnapshot[applied={T|T < 7299018141684989952}]
I20260620 20:20:51.029313 3580 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:32772:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 20:20:51.029635 3580 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:32772:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 20:20:51.033609 3580 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 301775c4958246bcb58f1e9a9a4546c0: Sending DeleteTablet for 3 replicas of tablet abc3a5c351914ae9b33ccd8216eb3cf9
I20260620 20:20:51.035223 3831 tablet_service.cc:1514] Processing DeleteTablet for tablet abc3a5c351914ae9b33ccd8216eb3cf9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:20:51 UTC) from {username='slave'} at 127.0.0.1:36738
I20260620 20:20:51.035437 3701 tablet_service.cc:1514] Processing DeleteTablet for tablet abc3a5c351914ae9b33ccd8216eb3cf9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:20:51 UTC) from {username='slave'} at 127.0.0.1:40412
I20260620 20:20:51.035537 3832 tablet_service.cc:1514] Processing DeleteTablet for tablet abc3a5c351914ae9b33ccd8216eb3cf9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:20:51 UTC) from {username='slave'} at 127.0.0.1:36738
I20260620 20:20:51.035919 3962 tablet_service.cc:1514] Processing DeleteTablet for tablet abc3a5c351914ae9b33ccd8216eb3cf9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:20:51 UTC) from {username='slave'} at 127.0.0.1:54564
I20260620 20:20:51.036547 17252 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu pid 3639
I20260620 20:20:51.046154 4070 tablet_replica.cc:330] T abc3a5c351914ae9b33ccd8216eb3cf9 P 8b72aba0078b42018e69e8ec17932cb7: stopping tablet replica
I20260620 20:20:51.046802 4070 raft_consensus.cc:2229] T abc3a5c351914ae9b33ccd8216eb3cf9 P 8b72aba0078b42018e69e8ec17932cb7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 20:20:51.047493 4070 raft_consensus.cc:2258] T abc3a5c351914ae9b33ccd8216eb3cf9 P 8b72aba0078b42018e69e8ec17932cb7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:20:51.048496 4069 tablet_replica.cc:330] T abc3a5c351914ae9b33ccd8216eb3cf9 P 4ac8c33a74774f709570c82c6fa95b97: stopping tablet replica
I20260620 20:20:51.048537 4068 tablet_replica.cc:330] T abc3a5c351914ae9b33ccd8216eb3cf9 P a94f66671bd948a2999c243799a9c873: stopping tablet replica
I20260620 20:20:51.048941 4068 raft_consensus.cc:2229] T abc3a5c351914ae9b33ccd8216eb3cf9 P a94f66671bd948a2999c243799a9c873 [term 1 LEADER]: Raft consensus shutting down.
I20260620 20:20:51.049616 4068 raft_consensus.cc:2258] T abc3a5c351914ae9b33ccd8216eb3cf9 P a94f66671bd948a2999c243799a9c873 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:20:51.050477 4069 raft_consensus.cc:2229] T abc3a5c351914ae9b33ccd8216eb3cf9 P 4ac8c33a74774f709570c82c6fa95b97 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 20:20:51.050997 4069 raft_consensus.cc:2258] T abc3a5c351914ae9b33ccd8216eb3cf9 P 4ac8c33a74774f709570c82c6fa95b97 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:20:51.049953 4070 ts_tablet_manager.cc:1905] T abc3a5c351914ae9b33ccd8216eb3cf9 P 8b72aba0078b42018e69e8ec17932cb7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:20:51.051575 4068 ts_tablet_manager.cc:1905] T abc3a5c351914ae9b33ccd8216eb3cf9 P a94f66671bd948a2999c243799a9c873: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:20:51.058034 4070 ts_tablet_manager.cc:1918] T abc3a5c351914ae9b33ccd8216eb3cf9 P 8b72aba0078b42018e69e8ec17932cb7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 20:20:51.055883 4040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:51.037790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40420 (local address 127.16.217.1:43063)
0620 20:20:51.037980 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0620 20:20:51.037986 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:20:51.043710 (+ 5724us) server_negotiation.cc:407] Connection header received
0620 20:20:51.043923 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:20:51.043929 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:20:51.044066 (+ 137us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:20:51.044172 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:20:51.048832 (+ 4660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:51.050058 (+ 1226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:51.051996 (+ 1938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:51.052657 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:51.052753 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:20:51.054213 (+ 1460us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:20:51.054224 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:20:51.054241 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:20:51.054949 (+ 708us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:20:51.055007 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:20:51.055034 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:20:51.055124 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0620 20:20:51.055389 (+ 265us) server_negotiation.cc:299] Negotiation successful
0620 20:20:51.055517 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 20:20:51.058620 4068 ts_tablet_manager.cc:1918] T abc3a5c351914ae9b33ccd8216eb3cf9 P a94f66671bd948a2999c243799a9c873: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 20:20:51.058782 4068 log.cc:1196] T abc3a5c351914ae9b33ccd8216eb3cf9 P a94f66671bd948a2999c243799a9c873: Deleting WAL directory at /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986536687152-17252-0/minicluster-data/ts-1/wal/wals/abc3a5c351914ae9b33ccd8216eb3cf9
I20260620 20:20:51.059098 3761 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:20:51.059325 4068 ts_tablet_manager.cc:1939] T abc3a5c351914ae9b33ccd8216eb3cf9 P a94f66671bd948a2999c243799a9c873: Deleting consensus metadata
I20260620 20:20:51.061527 3564 catalog_manager.cc:4841] TS a94f66671bd948a2999c243799a9c873 (127.16.217.2:35183): tablet abc3a5c351914ae9b33ccd8216eb3cf9 (table test-table [id=098b720ecd994921ac19a2ac0a1e3120]) successfully deleted
W20260620 20:20:51.062018 3564 catalog_manager.cc:4808] TS a94f66671bd948a2999c243799a9c873 (127.16.217.2:35183): delete failed for tablet abc3a5c351914ae9b33ccd8216eb3cf9 because the tablet was not found. No further retry: Not found: Tablet not found: abc3a5c351914ae9b33ccd8216eb3cf9
I20260620 20:20:51.065248 4070 log.cc:1196] T abc3a5c351914ae9b33ccd8216eb3cf9 P 8b72aba0078b42018e69e8ec17932cb7: Deleting WAL directory at /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986536687152-17252-0/minicluster-data/ts-2/wal/wals/abc3a5c351914ae9b33ccd8216eb3cf9
I20260620 20:20:51.065914 4070 ts_tablet_manager.cc:1939] T abc3a5c351914ae9b33ccd8216eb3cf9 P 8b72aba0078b42018e69e8ec17932cb7: Deleting consensus metadata
I20260620 20:20:51.068049 3564 catalog_manager.cc:4841] TS 8b72aba0078b42018e69e8ec17932cb7 (127.16.217.3:36525): tablet abc3a5c351914ae9b33ccd8216eb3cf9 (table test-table [id=098b720ecd994921ac19a2ac0a1e3120]) successfully deleted
I20260620 20:20:51.153092 17252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu with pid 3639
W20260620 20:20:51.178113 3564 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.16.217.1:43063 (error 108)
I20260620 20:20:51.178349 17252 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu pid 3770
I20260620 20:20:51.180270 4039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:51.179739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.1:43063 (local address 127.0.0.1:40434)
0620 20:20:51.179940 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0620 20:20:51.180069 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.217.1:43063: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260620 20:20:51.180526 3564 catalog_manager.cc:4584] TS 4ac8c33a74774f709570c82c6fa95b97 (127.16.217.1:43063): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet abc3a5c351914ae9b33ccd8216eb3cf9: Network error: Client connection negotiation failed: client connection to 127.16.217.1:43063: connect: Connection refused (error 111)
I20260620 20:20:51.186353 4043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:51.179155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36768 (local address 127.16.217.2:35183)
0620 20:20:51.179351 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0620 20:20:51.179358 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:20:51.179496 (+ 138us) server_negotiation.cc:407] Connection header received
0620 20:20:51.179681 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:20:51.179687 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:20:51.179790 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:20:51.179909 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:20:51.180562 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:51.181803 (+ 1241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:51.183320 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:51.183753 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:51.183893 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:20:51.184939 (+ 1046us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:20:51.184948 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:20:51.184962 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:20:51.185574 (+ 612us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:20:51.185627 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:20:51.185645 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:20:51.185766 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0620 20:20:51.185931 (+ 165us) server_negotiation.cc:299] Negotiation successful
0620 20:20:51.186026 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 20:20:51.187172 3892 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:20:51.234408 4039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:51.233828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.1:43063 (local address 127.0.0.1:40448)
0620 20:20:51.234105 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0620 20:20:51.234221 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.217.1:43063: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107}
I20260620 20:20:51.235600 4039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:51.235170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.1:43063 (local address 127.0.0.1:40452)
0620 20:20:51.235332 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0620 20:20:51.235435 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.217.1:43063: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260620 20:20:51.296337 4039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:51.295757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.1:43063 (local address 127.0.0.1:40458)
0620 20:20:51.296030 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0620 20:20:51.296163 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.217.1:43063: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
I20260620 20:20:51.297442 4039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:51.297063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.1:43063 (local address 127.0.0.1:40474)
0620 20:20:51.297212 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0620 20:20:51.297313 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.217.1:43063: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260620 20:20:51.306838 17252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu with pid 3770
I20260620 20:20:51.326220 17252 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu pid 3900
I20260620 20:20:51.333086 4042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:51.327013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54566 (local address 127.16.217.3:36525)
0620 20:20:51.327199 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0620 20:20:51.327207 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 20:20:51.327243 (+ 36us) server_negotiation.cc:407] Connection header received
0620 20:20:51.327436 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:20:51.327440 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:20:51.327514 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:20:51.327618 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:20:51.328015 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:51.328771 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:51.330136 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:51.330519 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:51.330649 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:20:51.331696 (+ 1047us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:20:51.331705 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:20:51.331719 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:20:51.332324 (+ 605us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:20:51.332377 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:20:51.332396 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:20:51.332516 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0620 20:20:51.332660 (+ 144us) server_negotiation.cc:299] Negotiation successful
0620 20:20:51.332778 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 20:20:51.333716 4022 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:20:51.378479 4039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:51.377909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.1:43063 (local address 127.0.0.1:40480)
0620 20:20:51.378179 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0620 20:20:51.378291 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.217.1:43063: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260620 20:20:51.379629 4039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:51.379161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.1:43063 (local address 127.0.0.1:40496)
0620 20:20:51.379365 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0620 20:20:51.379452 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.217.1:43063: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
I20260620 20:20:51.421609 17252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu with pid 3900
I20260620 20:20:51.439304 17252 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu pid 3549
I20260620 20:20:51.440353 3610 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:20:51.548624 17252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu with pid 3549
2026-06-20T20:20:51Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (2892 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986536687152-17252-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 20:20:51 dist-test-slave-2hpg krb5kdc[4079](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 20:20:51 dist-test-slave-2hpg krb5kdc[4079](info): set up 2 sockets
Jun 20 20:20:51 dist-test-slave-2hpg krb5kdc[4079](info): commencing operation
krb5kdc: starting...
W20260620 20:20:52.658474 17252 mini_kdc.cc:121] Time spent starting KDC: real 1.068s user 0.008s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986536687152-17252-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986536687152-17252-0/krb5kdc/test-user.keytab.
Jun 20 20:20:52 dist-test-slave-2hpg krb5kdc[4079](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986852, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T20:20:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T20:20:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.217.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.217.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.217.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.62.keytab.
Entry for principal kudu/127.16.217.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.62.keytab.
I20260620 20:20:52.898597 17252 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
/tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986536687152-17252-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986536687152-17252-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986536687152-17252-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986536687152-17252-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.217.62:35921
--webserver_interface=127.16.217.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.62.keytab
--principal=kudu/127.16.217.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.217.20:34723
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.217.62:35921
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986536687152-17252-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986536687152-17252-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986536687152-17252-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:20:53.228849 4092 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:20:53.229121 4092 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:20:53.229286 4092 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:20:53.236538 4092 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:20:53.236644 4092 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:20:53.236740 4092 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:20:53.236830 4092 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:20:53.236907 4092 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:20:53.246727 4092 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.217.20:34723
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986536687152-17252-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986536687152-17252-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.217.62:35921
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.62.keytab
--principal=kudu/127.16.217.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.217.62:35921
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986536687152-17252-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.217.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986536687152-17252-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:20:53.248015 4092 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:20:53.249292 4092 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:20:53.260586 4098 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:20:53.260650 4100 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:20:53.261117 4097 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:20:53.261467 4092 server_base.cc:1034] running on GCE node
I20260620 20:20:53.262327 4092 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:20:53.263767 4092 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:20:53.264999 4092 hybrid_clock.cc:648] HybridClock initialized: now 1781986853264936 us; error 52 us; skew 500 ppm
I20260620 20:20:53.265358 4092 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:20:53 dist-test-slave-2hpg krb5kdc[4079](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986853, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.217.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:20:53.268798 4092 init.cc:377] Logged in from keytab as kudu/127.16.217.62@KRBTEST.COM (short username kudu)
I20260620 20:20:53.270429 4092 webserver.cc:460] Webserver started at http://127.16.217.62:35685/ using document root <none> and password file <none>
I20260620 20:20:53.271190 4092 fs_manager.cc:359] Metadata directory not provided
I20260620 20:20:53.271306 4092 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:20:53.271608 4092 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:20:53.274778 4092 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986536687152-17252-0/minicluster-data/master-0/data/instance:
uuid: "9ff3106ab4874c828568957739cb3a8b"
format_stamp: "Formatted at 2026-06-20 20:20:53 on dist-test-slave-2hpg"
server_key: "1458bc08fb5a3f64345a6912285e431c"
server_key_iv: "0ff7109d19332fb5e0dbdaa3a02886f8"
server_key_version: "encryptionkey@0"
I20260620 20:20:53.275557 4092 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986536687152-17252-0/minicluster-data/master-0/wal/instance:
uuid: "9ff3106ab4874c828568957739cb3a8b"
format_stamp: "Formatted at 2026-06-20 20:20:53 on dist-test-slave-2hpg"
server_key: "1458bc08fb5a3f64345a6912285e431c"
server_key_iv: "0ff7109d19332fb5e0dbdaa3a02886f8"
server_key_version: "encryptionkey@0"
I20260620 20:20:53.281203 4092 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 20:20:53.284722 4107 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:20:53.286182 4092 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 20:20:53.286387 4092 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986536687152-17252-0/minicluster-data/master-0/data,/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986536687152-17252-0/minicluster-data/master-0/wal
uuid: "9ff3106ab4874c828568957739cb3a8b"
format_stamp: "Formatted at 2026-06-20 20:20:53 on dist-test-slave-2hpg"
server_key: "1458bc08fb5a3f64345a6912285e431c"
server_key_iv: "0ff7109d19332fb5e0dbdaa3a02886f8"
server_key_version: "encryptionkey@0"
I20260620 20:20:53.286625 4092 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986536687152-17252-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986536687152-17252-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986536687152-17252-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:20:53.313625 4092 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:20:53.318147 4092 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:20:53.318434 4092 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:20:53.336501 4092 rpc_server.cc:307] RPC server started. Bound to: 127.16.217.62:35921
I20260620 20:20:53.336515 4158 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.217.62:35921 every 8 connection(s)
I20260620 20:20:53.337880 4092 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986536687152-17252-0/minicluster-data/master-0/data/info.pb
I20260620 20:20:53.341516 4159 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:20:53.341956 17252 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu as pid 4092
I20260620 20:20:53.342132 17252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986536687152-17252-0/minicluster-data/master-0/wal/instance
I20260620 20:20:53.342545 17252 external_mini_cluster.cc:1428] Setting key 3e729622d170154e1e70433802746936
I20260620 20:20:53.355851 4159 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9ff3106ab4874c828568957739cb3a8b: Bootstrap starting.
Jun 20 20:20:53 dist-test-slave-2hpg krb5kdc[4079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986852, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.217.62@KRBTEST.COM
I20260620 20:20:53.359422 4159 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9ff3106ab4874c828568957739cb3a8b: Neither blocks nor log segments found. Creating new log.
I20260620 20:20:53.360807 4159 log.cc:824] T 00000000000000000000000000000000 P 9ff3106ab4874c828568957739cb3a8b: Log is configured to *not* fsync() on all Append() calls
I20260620 20:20:53.364360 4159 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9ff3106ab4874c828568957739cb3a8b: No bootstrap required, opened a new log
I20260620 20:20:53.367576 4162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:53.344418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55000 (local address 127.16.217.62:35921)
0620 20:20:53.345225 (+ 807us) server_negotiation.cc:206] Beginning negotiation
0620 20:20:53.345240 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 20:20:53.345288 (+ 48us) server_negotiation.cc:407] Connection header received
0620 20:20:53.348156 (+ 2868us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:20:53.348186 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:20:53.348745 (+ 559us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:20:53.349202 (+ 457us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:20:53.350598 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:53.352606 (+ 2008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:53.354002 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:53.355037 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:53.355224 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:20:53.359684 (+ 4460us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:20:53.359692 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:20:53.359705 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:20:53.359744 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:20:53.363265 (+ 3521us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:20:53.364212 (+ 947us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:20:53.364218 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:20:53.364223 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:20:53.364383 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:20:53.364990 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:20:53.364994 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:20:53.364996 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:20:53.365495 (+ 499us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:20:53.365734 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0620 20:20:53.366209 (+ 475us) server_negotiation.cc:299] Negotiation successful
0620 20:20:53.366545 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":551,"thread_start_us":322,"threads_started":1}
I20260620 20:20:53.373793 4159 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9ff3106ab4874c828568957739cb3a8b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ff3106ab4874c828568957739cb3a8b" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 35921 } }
I20260620 20:20:53.374320 4159 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9ff3106ab4874c828568957739cb3a8b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:20:53.374440 4159 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9ff3106ab4874c828568957739cb3a8b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ff3106ab4874c828568957739cb3a8b, State: Initialized, Role: FOLLOWER
I20260620 20:20:53.375020 4159 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9ff3106ab4874c828568957739cb3a8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ff3106ab4874c828568957739cb3a8b" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 35921 } }
I20260620 20:20:53.375344 4159 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9ff3106ab4874c828568957739cb3a8b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 20:20:53.375535 4159 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9ff3106ab4874c828568957739cb3a8b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 20:20:53.375765 4159 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9ff3106ab4874c828568957739cb3a8b [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:20:53.378736 4159 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9ff3106ab4874c828568957739cb3a8b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ff3106ab4874c828568957739cb3a8b" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 35921 } }
I20260620 20:20:53.379180 4159 leader_election.cc:302] T 00000000000000000000000000000000 P 9ff3106ab4874c828568957739cb3a8b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9ff3106ab4874c828568957739cb3a8b; no voters:
I20260620 20:20:53.379776 4159 leader_election.cc:288] T 00000000000000000000000000000000 P 9ff3106ab4874c828568957739cb3a8b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 20:20:53.379899 4164 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9ff3106ab4874c828568957739cb3a8b [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:20:53.380885 4164 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9ff3106ab4874c828568957739cb3a8b [term 1 LEADER]: Becoming Leader. State: Replica: 9ff3106ab4874c828568957739cb3a8b, State: Running, Role: LEADER
I20260620 20:20:53.381616 4164 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9ff3106ab4874c828568957739cb3a8b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ff3106ab4874c828568957739cb3a8b" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 35921 } }
I20260620 20:20:53.382193 4159 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9ff3106ab4874c828568957739cb3a8b [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:20:53.384740 4166 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ff3106ab4874c828568957739cb3a8b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9ff3106ab4874c828568957739cb3a8b. Latest consensus state: current_term: 1 leader_uuid: "9ff3106ab4874c828568957739cb3a8b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ff3106ab4874c828568957739cb3a8b" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 35921 } } }
I20260620 20:20:53.385682 4166 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ff3106ab4874c828568957739cb3a8b [sys.catalog]: This master's current role is: LEADER
I20260620 20:20:53.385228 4165 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ff3106ab4874c828568957739cb3a8b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9ff3106ab4874c828568957739cb3a8b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ff3106ab4874c828568957739cb3a8b" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 35921 } } }
I20260620 20:20:53.385949 4165 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ff3106ab4874c828568957739cb3a8b [sys.catalog]: This master's current role is: LEADER
I20260620 20:20:53.387924 4172 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 20:20:53.393260 4172 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 20:20:53.402652 4172 catalog_manager.cc:1347] Generated new cluster ID: a7ee319b81ea4527b433cee82fdf1800
I20260620 20:20:53.402782 4172 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 20:20:53.427685 4172 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 20:20:53.429036 4172 catalog_manager.cc:1504] Loading token signing keys...
I20260620 20:20:53.436820 4172 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9ff3106ab4874c828568957739cb3a8b: Generated new TSK 0
I20260620 20:20:53.437665 4172 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.217.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.217.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.217.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.1.keytab.
Entry for principal kudu/127.16.217.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.1.keytab.
I20260620 20:20:53.507975 17252 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
/tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986536687152-17252-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986536687152-17252-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986536687152-17252-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986536687152-17252-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.217.1:0
--local_ip_for_outbound_sockets=127.16.217.1
--webserver_interface=127.16.217.1
--webserver_port=0
--tserver_master_addrs=127.16.217.62:35921
--keytab_file=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.1.keytab
--principal=kudu/127.16.217.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.217.20:34723
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986536687152-17252-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986536687152-17252-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986536687152-17252-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:20:53.837746 4185 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:20:53.838092 4185 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:20:53.838263 4185 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:20:53.845467 4185 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:20:53.845613 4185 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:20:53.845798 4185 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.217.1
I20260620 20:20:53.856046 4185 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.217.20:34723
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986536687152-17252-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986536687152-17252-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.1.keytab
--principal=kudu/127.16.217.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.217.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986536687152-17252-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.217.1
--webserver_port=0
--tserver_master_addrs=127.16.217.62:35921
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.217.1
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986536687152-17252-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:20:53.857335 4185 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:20:53.858740 4185 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:20:53.871984 4190 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:20:53.871956 4193 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:20:53.872115 4191 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:20:53.872689 4185 server_base.cc:1034] running on GCE node
I20260620 20:20:53.873311 4185 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:20:53.874293 4185 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:20:53.875531 4185 hybrid_clock.cc:648] HybridClock initialized: now 1781986853875480 us; error 28 us; skew 500 ppm
I20260620 20:20:53.875851 4185 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:20:53 dist-test-slave-2hpg krb5kdc[4079](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986853, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.217.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:20:53.879128 4185 init.cc:377] Logged in from keytab as kudu/127.16.217.1@KRBTEST.COM (short username kudu)
I20260620 20:20:53.880549 4185 webserver.cc:460] Webserver started at http://127.16.217.1:34585/ using document root <none> and password file <none>
I20260620 20:20:53.881227 4185 fs_manager.cc:359] Metadata directory not provided
I20260620 20:20:53.881385 4185 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:20:53.881703 4185 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:20:53.884822 4185 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986536687152-17252-0/minicluster-data/ts-0/data/instance:
uuid: "5ce0cbb9797a42399cee61c9291d59e1"
format_stamp: "Formatted at 2026-06-20 20:20:53 on dist-test-slave-2hpg"
server_key: "b3d083a508722e86341e397acbe5243e"
server_key_iv: "413d823f9aad42ee26d38952c86930bc"
server_key_version: "encryptionkey@0"
I20260620 20:20:53.885581 4185 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986536687152-17252-0/minicluster-data/ts-0/wal/instance:
uuid: "5ce0cbb9797a42399cee61c9291d59e1"
format_stamp: "Formatted at 2026-06-20 20:20:53 on dist-test-slave-2hpg"
server_key: "b3d083a508722e86341e397acbe5243e"
server_key_iv: "413d823f9aad42ee26d38952c86930bc"
server_key_version: "encryptionkey@0"
I20260620 20:20:53.890908 4185 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.002s
I20260620 20:20:53.894858 4200 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:20:53.896139 4185 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 20:20:53.896318 4185 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986536687152-17252-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986536687152-17252-0/minicluster-data/ts-0/wal
uuid: "5ce0cbb9797a42399cee61c9291d59e1"
format_stamp: "Formatted at 2026-06-20 20:20:53 on dist-test-slave-2hpg"
server_key: "b3d083a508722e86341e397acbe5243e"
server_key_iv: "413d823f9aad42ee26d38952c86930bc"
server_key_version: "encryptionkey@0"
I20260620 20:20:53.896538 4185 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986536687152-17252-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986536687152-17252-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986536687152-17252-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:20:53.919642 4185 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:20:53.924814 4185 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:20:53.925171 4185 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:20:53.926221 4185 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:20:53.927561 4185 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:20:53.927673 4185 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:20:53.927824 4185 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:20:53.927893 4185 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:20:53.960350 4185 rpc_server.cc:307] RPC server started. Bound to: 127.16.217.1:41283
I20260620 20:20:53.960439 4312 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.217.1:41283 every 8 connection(s)
I20260620 20:20:53.961675 4185 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986536687152-17252-0/minicluster-data/ts-0/data/info.pb
I20260620 20:20:53.964735 17252 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu as pid 4185
I20260620 20:20:53.964936 17252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986536687152-17252-0/minicluster-data/ts-0/wal/instance
I20260620 20:20:53.965356 17252 external_mini_cluster.cc:1428] Setting key 99faa98f225804ac1e341350e1cf0e14
Jun 20 20:20:53 dist-test-slave-2hpg krb5kdc[4079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986853, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.217.1@KRBTEST.COM for kudu/127.16.217.62@KRBTEST.COM
I20260620 20:20:53.982343 4315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:53.964131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.217.1:38169 (local address 127.16.217.62:35921)
0620 20:20:53.964688 (+ 557us) server_negotiation.cc:206] Beginning negotiation
0620 20:20:53.964696 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 20:20:53.965750 (+ 1054us) server_negotiation.cc:407] Connection header received
0620 20:20:53.967861 (+ 2111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:20:53.967866 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:20:53.967950 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:20:53.968082 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:20:53.969568 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:53.970740 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:53.972277 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:53.973043 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:53.973193 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:20:53.976634 (+ 3441us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:20:53.976639 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:20:53.976643 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:20:53.976673 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:20:53.979161 (+ 2488us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:20:53.980045 (+ 884us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:20:53.980049 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:20:53.980061 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:20:53.980171 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:20:53.980629 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:20:53.980632 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:20:53.980634 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:20:53.980867 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:20:53.980972 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0620 20:20:53.981820 (+ 848us) server_negotiation.cc:299] Negotiation successful
0620 20:20:53.982072 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":424,"thread_start_us":275,"threads_started":1}
I20260620 20:20:53.983857 4316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:53.964727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:35921 (local address 127.16.217.1:38169)
0620 20:20:53.965578 (+ 851us) negotiation.cc:107] Waiting for socket to connect
0620 20:20:53.965623 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 20:20:53.967614 (+ 1991us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:20:53.968265 (+ 651us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:20:53.968274 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:20:53.968840 (+ 566us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:20:53.969398 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:20:53.969416 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:20:53.970909 (+ 1493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:20:53.970913 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:20:53.972131 (+ 1218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:20:53.972143 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:20:53.973262 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:20:53.973267 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:20:53.973463 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:20:53.974177 (+ 714us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:20:53.974199 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:20:53.976439 (+ 2240us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:20:53.979294 (+ 2855us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:20:53.979306 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:20:53.979324 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:20:53.979938 (+ 614us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:20:53.980272 (+ 334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:20:53.980279 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:20:53.980285 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:20:53.980521 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:20:53.980994 (+ 473us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:20:53.981004 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:20:53.981455 (+ 451us) client_negotiation.cc:769] Sending connection context
0620 20:20:53.981746 (+ 291us) client_negotiation.cc:240] Negotiation successful
0620 20:20:53.982347 (+ 601us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":575,"thread_start_us":387,"threads_started":1}
I20260620 20:20:53.985460 4313 heartbeater.cc:347] Connected to a master server at 127.16.217.62:35921
I20260620 20:20:53.985785 4313 heartbeater.cc:464] Registering TS with master...
I20260620 20:20:53.986582 4313 heartbeater.cc:511] Master 127.16.217.62:35921 requested a full tablet report, sending...
I20260620 20:20:53.988747 4124 ts_manager.cc:194] Registered new tserver with Master: 5ce0cbb9797a42399cee61c9291d59e1 (127.16.217.1:41283)
I20260620 20:20:53.990376 4124 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.217.1@KRBTEST.COM'} at 127.16.217.1:38169
WARNING: no policy specified for kudu/127.16.217.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.217.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.217.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.2.keytab.
Entry for principal kudu/127.16.217.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.2.keytab.
I20260620 20:20:54.024802 17252 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
/tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986536687152-17252-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986536687152-17252-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986536687152-17252-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986536687152-17252-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.217.2:0
--local_ip_for_outbound_sockets=127.16.217.2
--webserver_interface=127.16.217.2
--webserver_port=0
--tserver_master_addrs=127.16.217.62:35921
--keytab_file=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.2.keytab
--principal=kudu/127.16.217.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.217.20:34723
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986536687152-17252-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986536687152-17252-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986536687152-17252-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:20:54.353979 4319 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:20:54.354303 4319 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:20:54.354476 4319 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:20:54.361909 4319 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:20:54.362082 4319 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:20:54.362257 4319 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.217.2
I20260620 20:20:54.372189 4319 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.217.20:34723
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986536687152-17252-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986536687152-17252-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.2.keytab
--principal=kudu/127.16.217.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.217.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986536687152-17252-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.217.2
--webserver_port=0
--tserver_master_addrs=127.16.217.62:35921
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.217.2
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986536687152-17252-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:20:54.373559 4319 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:20:54.374946 4319 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:20:54.388481 4324 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:20:54.388638 4325 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:20:54.389161 4327 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:20:54.390130 4319 server_base.cc:1034] running on GCE node
I20260620 20:20:54.390885 4319 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:20:54.391911 4319 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:20:54.393167 4319 hybrid_clock.cc:648] HybridClock initialized: now 1781986854393112 us; error 44 us; skew 500 ppm
I20260620 20:20:54.393514 4319 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:20:54 dist-test-slave-2hpg krb5kdc[4079](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986854, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.217.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:20:54.396819 4319 init.cc:377] Logged in from keytab as kudu/127.16.217.2@KRBTEST.COM (short username kudu)
I20260620 20:20:54.398501 4319 webserver.cc:460] Webserver started at http://127.16.217.2:40043/ using document root <none> and password file <none>
I20260620 20:20:54.399235 4319 fs_manager.cc:359] Metadata directory not provided
I20260620 20:20:54.399384 4319 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:20:54.399718 4319 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:20:54.402770 4319 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986536687152-17252-0/minicluster-data/ts-1/data/instance:
uuid: "a083da43d71847bda8d84f08dfb866be"
format_stamp: "Formatted at 2026-06-20 20:20:54 on dist-test-slave-2hpg"
server_key: "d18160f4e24541228426b84ad919822a"
server_key_iv: "cdfa4a2c4071eb98f66b87b9ed7143c7"
server_key_version: "encryptionkey@0"
I20260620 20:20:54.403568 4319 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986536687152-17252-0/minicluster-data/ts-1/wal/instance:
uuid: "a083da43d71847bda8d84f08dfb866be"
format_stamp: "Formatted at 2026-06-20 20:20:54 on dist-test-slave-2hpg"
server_key: "d18160f4e24541228426b84ad919822a"
server_key_iv: "cdfa4a2c4071eb98f66b87b9ed7143c7"
server_key_version: "encryptionkey@0"
I20260620 20:20:54.409047 4319 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 20:20:54.412760 4334 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:20:54.414283 4319 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 20:20:54.414484 4319 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986536687152-17252-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986536687152-17252-0/minicluster-data/ts-1/wal
uuid: "a083da43d71847bda8d84f08dfb866be"
format_stamp: "Formatted at 2026-06-20 20:20:54 on dist-test-slave-2hpg"
server_key: "d18160f4e24541228426b84ad919822a"
server_key_iv: "cdfa4a2c4071eb98f66b87b9ed7143c7"
server_key_version: "encryptionkey@0"
I20260620 20:20:54.414711 4319 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986536687152-17252-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986536687152-17252-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986536687152-17252-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:20:54.437737 4319 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:20:54.441958 4319 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:20:54.442286 4319 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:20:54.443284 4319 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:20:54.444624 4319 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:20:54.444736 4319 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:20:54.444895 4319 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:20:54.444967 4319 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:20:54.478935 4319 rpc_server.cc:307] RPC server started. Bound to: 127.16.217.2:34483
I20260620 20:20:54.479027 4446 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.217.2:34483 every 8 connection(s)
I20260620 20:20:54.480314 4319 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986536687152-17252-0/minicluster-data/ts-1/data/info.pb
I20260620 20:20:54.489279 17252 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu as pid 4319
I20260620 20:20:54.489466 17252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986536687152-17252-0/minicluster-data/ts-1/wal/instance
I20260620 20:20:54.489959 17252 external_mini_cluster.cc:1428] Setting key fbab4adec86f6b08ae0c9260f333a800
Jun 20 20:20:54 dist-test-slave-2hpg krb5kdc[4079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986854, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.217.2@KRBTEST.COM for kudu/127.16.217.62@KRBTEST.COM
I20260620 20:20:54.502949 4449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:54.483087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.217.2:54029 (local address 127.16.217.62:35921)
0620 20:20:54.483557 (+ 470us) server_negotiation.cc:206] Beginning negotiation
0620 20:20:54.483561 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:20:54.484526 (+ 965us) server_negotiation.cc:407] Connection header received
0620 20:20:54.486444 (+ 1918us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:20:54.486449 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:20:54.486545 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:20:54.486662 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:20:54.488203 (+ 1541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:54.489003 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:54.491551 (+ 2548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:54.492123 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:54.492255 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:20:54.495719 (+ 3464us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:20:54.495728 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:20:54.495736 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:20:54.495780 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:20:54.499545 (+ 3765us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:20:54.500403 (+ 858us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:20:54.500411 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:20:54.500417 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:20:54.500580 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:20:54.501141 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:20:54.501146 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:20:54.501149 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:20:54.501482 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:20:54.501606 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0620 20:20:54.502518 (+ 912us) server_negotiation.cc:299] Negotiation successful
0620 20:20:54.502695 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":351,"thread_start_us":255,"threads_started":1}
I20260620 20:20:54.504492 4450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:54.483686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:35921 (local address 127.16.217.2:54029)
0620 20:20:54.484360 (+ 674us) negotiation.cc:107] Waiting for socket to connect
0620 20:20:54.484402 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 20:20:54.486176 (+ 1774us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:20:54.486903 (+ 727us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:20:54.486912 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:20:54.487434 (+ 522us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:20:54.488024 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:20:54.488042 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:20:54.489174 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:20:54.489179 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:20:54.490826 (+ 1647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:20:54.490839 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:20:54.492335 (+ 1496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:20:54.492341 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:20:54.492543 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:20:54.493241 (+ 698us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:20:54.493263 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:20:54.495474 (+ 2211us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:20:54.499698 (+ 4224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:20:54.499705 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:20:54.499718 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:20:54.500256 (+ 538us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:20:54.500719 (+ 463us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:20:54.500726 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:20:54.500732 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:20:54.500986 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:20:54.501618 (+ 632us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:20:54.501629 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:20:54.502122 (+ 493us) client_negotiation.cc:769] Sending connection context
0620 20:20:54.502423 (+ 301us) client_negotiation.cc:240] Negotiation successful
0620 20:20:54.503002 (+ 579us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":453,"thread_start_us":256,"threads_started":1}
I20260620 20:20:54.506008 4447 heartbeater.cc:347] Connected to a master server at 127.16.217.62:35921
I20260620 20:20:54.506341 4447 heartbeater.cc:464] Registering TS with master...
I20260620 20:20:54.507125 4447 heartbeater.cc:511] Master 127.16.217.62:35921 requested a full tablet report, sending...
I20260620 20:20:54.508615 4124 ts_manager.cc:194] Registered new tserver with Master: a083da43d71847bda8d84f08dfb866be (127.16.217.2:34483)
I20260620 20:20:54.509426 4124 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.217.2@KRBTEST.COM'} at 127.16.217.2:54029
WARNING: no policy specified for kudu/127.16.217.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.217.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.217.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.3.keytab.
Entry for principal kudu/127.16.217.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.3.keytab.
I20260620 20:20:54.547585 17252 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
/tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986536687152-17252-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986536687152-17252-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986536687152-17252-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986536687152-17252-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.217.3:0
--local_ip_for_outbound_sockets=127.16.217.3
--webserver_interface=127.16.217.3
--webserver_port=0
--tserver_master_addrs=127.16.217.62:35921
--keytab_file=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.3.keytab
--principal=kudu/127.16.217.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.217.20:34723
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986536687152-17252-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986536687152-17252-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986536687152-17252-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:20:54.878099 4453 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:20:54.878433 4453 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:20:54.878615 4453 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:20:54.886106 4453 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:20:54.886248 4453 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:20:54.886389 4453 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.217.3
I20260620 20:20:54.896238 4453 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.217.20:34723
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986536687152-17252-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986536687152-17252-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.3.keytab
--principal=kudu/127.16.217.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.217.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986536687152-17252-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.217.3
--webserver_port=0
--tserver_master_addrs=127.16.217.62:35921
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.217.3
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986536687152-17252-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:20:54.897466 4453 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:20:54.899021 4453 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:20:54.912451 4459 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:20:54.912752 4458 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:20:54.912793 4461 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:20:54.913822 4453 server_base.cc:1034] running on GCE node
I20260620 20:20:54.914510 4453 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:20:54.915503 4453 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:20:54.916726 4453 hybrid_clock.cc:648] HybridClock initialized: now 1781986854916676 us; error 61 us; skew 500 ppm
I20260620 20:20:54.917040 4453 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:20:54 dist-test-slave-2hpg krb5kdc[4079](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986854, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.217.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:20:54.920436 4453 init.cc:377] Logged in from keytab as kudu/127.16.217.3@KRBTEST.COM (short username kudu)
I20260620 20:20:54.921825 4453 webserver.cc:460] Webserver started at http://127.16.217.3:35455/ using document root <none> and password file <none>
I20260620 20:20:54.922540 4453 fs_manager.cc:359] Metadata directory not provided
I20260620 20:20:54.922683 4453 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:20:54.922992 4453 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:20:54.926187 4453 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986536687152-17252-0/minicluster-data/ts-2/data/instance:
uuid: "73c7ac4da1144abeaac551bc31c42ec1"
format_stamp: "Formatted at 2026-06-20 20:20:54 on dist-test-slave-2hpg"
server_key: "bdb8a51083d41885027304d3584ec485"
server_key_iv: "df384061a9f0eba75d75a312db9bfee1"
server_key_version: "encryptionkey@0"
I20260620 20:20:54.927086 4453 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986536687152-17252-0/minicluster-data/ts-2/wal/instance:
uuid: "73c7ac4da1144abeaac551bc31c42ec1"
format_stamp: "Formatted at 2026-06-20 20:20:54 on dist-test-slave-2hpg"
server_key: "bdb8a51083d41885027304d3584ec485"
server_key_iv: "df384061a9f0eba75d75a312db9bfee1"
server_key_version: "encryptionkey@0"
I20260620 20:20:54.935392 4453 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.006s sys 0.001s
I20260620 20:20:54.938825 4468 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:20:54.940143 4453 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 20:20:54.940310 4453 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986536687152-17252-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986536687152-17252-0/minicluster-data/ts-2/wal
uuid: "73c7ac4da1144abeaac551bc31c42ec1"
format_stamp: "Formatted at 2026-06-20 20:20:54 on dist-test-slave-2hpg"
server_key: "bdb8a51083d41885027304d3584ec485"
server_key_iv: "df384061a9f0eba75d75a312db9bfee1"
server_key_version: "encryptionkey@0"
I20260620 20:20:54.940497 4453 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986536687152-17252-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986536687152-17252-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986536687152-17252-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:20:54.976981 4453 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:20:54.981128 4453 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:20:54.981432 4453 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:20:54.982501 4453 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:20:54.984022 4453 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:20:54.984143 4453 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:20:54.984298 4453 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:20:54.984366 4453 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:20:54.993389 4313 heartbeater.cc:503] Master 127.16.217.62:35921 was elected leader, sending a full tablet report...
I20260620 20:20:55.018941 4453 rpc_server.cc:307] RPC server started. Bound to: 127.16.217.3:42797
I20260620 20:20:55.018998 4580 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.217.3:42797 every 8 connection(s)
I20260620 20:20:55.020299 4453 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986536687152-17252-0/minicluster-data/ts-2/data/info.pb
I20260620 20:20:55.021602 17252 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu as pid 4453
I20260620 20:20:55.021809 17252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986536687152-17252-0/minicluster-data/ts-2/wal/instance
I20260620 20:20:55.022276 17252 external_mini_cluster.cc:1428] Setting key 97928f3aa9fe32af28592ef97264eeaf
Jun 20 20:20:55 dist-test-slave-2hpg krb5kdc[4079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986854, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.217.3@KRBTEST.COM for kudu/127.16.217.62@KRBTEST.COM
I20260620 20:20:55.041600 4583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:55.022955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.217.3:49721 (local address 127.16.217.62:35921)
0620 20:20:55.023468 (+ 513us) server_negotiation.cc:206] Beginning negotiation
0620 20:20:55.023474 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:20:55.024578 (+ 1104us) server_negotiation.cc:407] Connection header received
0620 20:20:55.026678 (+ 2100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:20:55.026683 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:20:55.026767 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:20:55.026888 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:20:55.028357 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:55.029185 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:55.030765 (+ 1580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:55.031168 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:55.031295 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:20:55.034647 (+ 3352us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:20:55.034653 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:20:55.034656 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:20:55.034686 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:20:55.038257 (+ 3571us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:20:55.039054 (+ 797us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:20:55.039060 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:20:55.039064 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:20:55.039202 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:20:55.039684 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:20:55.039689 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:20:55.039695 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:20:55.039989 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:20:55.040173 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0620 20:20:55.040904 (+ 731us) server_negotiation.cc:299] Negotiation successful
0620 20:20:55.041143 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":374,"thread_start_us":249,"threads_started":1}
I20260620 20:20:55.042658 4584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:55.023631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:35921 (local address 127.16.217.3:49721)
0620 20:20:55.024436 (+ 805us) negotiation.cc:107] Waiting for socket to connect
0620 20:20:55.024473 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 20:20:55.026417 (+ 1944us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:20:55.027123 (+ 706us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:20:55.027131 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:20:55.027631 (+ 500us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:20:55.028177 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:20:55.028193 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:20:55.029337 (+ 1144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:20:55.029344 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:20:55.030616 (+ 1272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:20:55.030628 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:20:55.031388 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:20:55.031395 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:20:55.031630 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:20:55.032241 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:20:55.032262 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:20:55.034452 (+ 2190us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:20:55.038407 (+ 3955us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:20:55.038417 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:20:55.038442 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:20:55.038910 (+ 468us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:20:55.039319 (+ 409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:20:55.039325 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:20:55.039329 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:20:55.039532 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:20:55.040136 (+ 604us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:20:55.040144 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:20:55.040507 (+ 363us) client_negotiation.cc:769] Sending connection context
0620 20:20:55.040793 (+ 286us) client_negotiation.cc:240] Negotiation successful
0620 20:20:55.041366 (+ 573us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":544,"thread_start_us":341,"threads_started":1}
I20260620 20:20:55.043989 4581 heartbeater.cc:347] Connected to a master server at 127.16.217.62:35921
I20260620 20:20:55.044345 4581 heartbeater.cc:464] Registering TS with master...
I20260620 20:20:55.045209 4581 heartbeater.cc:511] Master 127.16.217.62:35921 requested a full tablet report, sending...
I20260620 20:20:55.046702 4124 ts_manager.cc:194] Registered new tserver with Master: 73c7ac4da1144abeaac551bc31c42ec1 (127.16.217.3:42797)
I20260620 20:20:55.047570 4124 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.217.3@KRBTEST.COM'} at 127.16.217.3:49721
I20260620 20:20:55.049911 17252 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 20:20:55.065546 4583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:55.053943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55008 (local address 127.16.217.62:35921)
0620 20:20:55.054141 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0620 20:20:55.054149 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 20:20:55.054562 (+ 413us) server_negotiation.cc:407] Connection header received
0620 20:20:55.054709 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:20:55.054713 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:20:55.054769 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:20:55.054844 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:20:55.056168 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:55.056864 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:55.058140 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:55.058560 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:55.058653 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:20:55.060817 (+ 2164us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:20:55.060821 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:20:55.060824 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:20:55.060851 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:20:55.063320 (+ 2469us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:20:55.063927 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:20:55.063930 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:20:55.063933 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:20:55.064045 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:20:55.064333 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:20:55.064336 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:20:55.064338 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:20:55.064636 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:20:55.064809 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0620 20:20:55.065066 (+ 257us) server_negotiation.cc:299] Negotiation successful
0620 20:20:55.065218 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 20:20:55.069358 4124 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55008:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 20:20:55.076224 4124 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 20:20:55.098661 4594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:55.088317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.3:42797 (local address 127.0.0.1:33336)
0620 20:20:55.089113 (+ 796us) negotiation.cc:107] Waiting for socket to connect
0620 20:20:55.089170 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0620 20:20:55.089362 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:20:55.090300 (+ 938us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:20:55.090307 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:20:55.090338 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:20:55.091940 (+ 1602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:20:55.091954 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:20:55.094034 (+ 2080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:20:55.094040 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:20:55.096333 (+ 2293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:20:55.096345 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:20:55.097610 (+ 1265us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:20:55.097616 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:20:55.097825 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:20:55.097927 (+ 102us) client_negotiation.cc:769] Sending connection context
0620 20:20:55.098066 (+ 139us) client_negotiation.cc:240] Negotiation successful
0620 20:20:55.098250 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":284,"threads_started":1}
I20260620 20:20:55.098631 4595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:55.088595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33336 (local address 127.16.217.3:42797)
0620 20:20:55.089504 (+ 909us) server_negotiation.cc:206] Beginning negotiation
0620 20:20:55.089514 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 20:20:55.089548 (+ 34us) server_negotiation.cc:407] Connection header received
0620 20:20:55.089658 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:20:55.089665 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:20:55.089949 (+ 284us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:20:55.090150 (+ 201us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:20:55.092165 (+ 2015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:55.093841 (+ 1676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:55.096481 (+ 2640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:55.097380 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:55.097524 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:20:55.097770 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0620 20:20:55.098126 (+ 356us) server_negotiation.cc:299] Negotiation successful
0620 20:20:55.098287 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":750,"thread_start_us":631,"threads_started":1}
I20260620 20:20:55.100854 4596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:55.089731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.1:41283 (local address 127.0.0.1:34994)
0620 20:20:55.091487 (+ 1756us) negotiation.cc:107] Waiting for socket to connect
0620 20:20:55.091517 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0620 20:20:55.091661 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:20:55.092476 (+ 815us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:20:55.092481 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:20:55.092509 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:20:55.092945 (+ 436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:20:55.092958 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:20:55.094811 (+ 1853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:20:55.094819 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:20:55.097506 (+ 2687us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:20:55.097525 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:20:55.099833 (+ 2308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:20:55.099840 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:20:55.100038 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:20:55.100075 (+ 37us) client_negotiation.cc:769] Sending connection context
0620 20:20:55.100225 (+ 150us) client_negotiation.cc:240] Negotiation successful
0620 20:20:55.100424 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":677,"mutex_wait_us":56,"thread_start_us":1157,"threads_started":1}
I20260620 20:20:55.100806 4597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:55.090201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34994 (local address 127.16.217.1:41283)
0620 20:20:55.091180 (+ 979us) server_negotiation.cc:206] Beginning negotiation
0620 20:20:55.091188 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 20:20:55.091811 (+ 623us) server_negotiation.cc:407] Connection header received
0620 20:20:55.091934 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:20:55.091942 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:20:55.092186 (+ 244us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:20:55.093202 (+ 1016us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:20:55.093427 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:55.094667 (+ 1240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:55.097715 (+ 3048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:55.099332 (+ 1617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:55.099460 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:20:55.099694 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0620 20:20:55.100281 (+ 587us) server_negotiation.cc:299] Negotiation successful
0620 20:20:55.100447 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":814,"thread_start_us":630,"threads_started":1}
I20260620 20:20:55.102944 4598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:55.090733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59218 (local address 127.16.217.2:34483)
0620 20:20:55.091317 (+ 584us) server_negotiation.cc:206] Beginning negotiation
0620 20:20:55.091328 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 20:20:55.092173 (+ 845us) server_negotiation.cc:407] Connection header received
0620 20:20:55.092461 (+ 288us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:20:55.092470 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:20:55.092697 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:20:55.092839 (+ 142us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:20:55.093463 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:55.094921 (+ 1458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:55.097716 (+ 2795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:55.099298 (+ 1582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:55.099458 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:20:55.099658 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0620 20:20:55.102475 (+ 2817us) server_negotiation.cc:299] Negotiation successful
0620 20:20:55.102645 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":358,"thread_start_us":430,"threads_started":1}
I20260620 20:20:55.102998 4599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:55.090304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.2:34483 (local address 127.0.0.1:59218)
0620 20:20:55.092061 (+ 1757us) negotiation.cc:107] Waiting for socket to connect
0620 20:20:55.092087 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0620 20:20:55.092273 (+ 186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:20:55.093052 (+ 779us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:20:55.093056 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:20:55.093075 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:20:55.093314 (+ 239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:20:55.093320 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:20:55.095082 (+ 1762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:20:55.095089 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:20:55.097507 (+ 2418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:20:55.097521 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:20:55.102055 (+ 4534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:20:55.102063 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:20:55.102266 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:20:55.102303 (+ 37us) client_negotiation.cc:769] Sending connection context
0620 20:20:55.102418 (+ 115us) client_negotiation.cc:240] Negotiation successful
0620 20:20:55.102569 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1637,"mutex_wait_us":72,"thread_start_us":1459,"threads_started":1}
I20260620 20:20:55.103238 4516 tablet_service.cc:1467] Processing CreateTablet for tablet 57a20b63b778418d9e3d2b7ef7499f0d (DEFAULT_TABLE table=test-table [id=4f413f8a26de449387d5163b25d85bbe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:20:55.104679 4516 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57a20b63b778418d9e3d2b7ef7499f0d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:20:55.104635 4248 tablet_service.cc:1467] Processing CreateTablet for tablet 57a20b63b778418d9e3d2b7ef7499f0d (DEFAULT_TABLE table=test-table [id=4f413f8a26de449387d5163b25d85bbe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:20:55.106112 4248 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57a20b63b778418d9e3d2b7ef7499f0d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:20:55.106576 4382 tablet_service.cc:1467] Processing CreateTablet for tablet 57a20b63b778418d9e3d2b7ef7499f0d (DEFAULT_TABLE table=test-table [id=4f413f8a26de449387d5163b25d85bbe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:20:55.107800 4382 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57a20b63b778418d9e3d2b7ef7499f0d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:20:55.117585 4600 tablet_bootstrap.cc:492] T 57a20b63b778418d9e3d2b7ef7499f0d P 5ce0cbb9797a42399cee61c9291d59e1: Bootstrap starting.
I20260620 20:20:55.122210 4600 tablet_bootstrap.cc:654] T 57a20b63b778418d9e3d2b7ef7499f0d P 5ce0cbb9797a42399cee61c9291d59e1: Neither blocks nor log segments found. Creating new log.
I20260620 20:20:55.122234 4602 tablet_bootstrap.cc:492] T 57a20b63b778418d9e3d2b7ef7499f0d P a083da43d71847bda8d84f08dfb866be: Bootstrap starting.
I20260620 20:20:55.123754 4600 log.cc:824] T 57a20b63b778418d9e3d2b7ef7499f0d P 5ce0cbb9797a42399cee61c9291d59e1: Log is configured to *not* fsync() on all Append() calls
I20260620 20:20:55.125181 4601 tablet_bootstrap.cc:492] T 57a20b63b778418d9e3d2b7ef7499f0d P 73c7ac4da1144abeaac551bc31c42ec1: Bootstrap starting.
I20260620 20:20:55.126134 4602 tablet_bootstrap.cc:654] T 57a20b63b778418d9e3d2b7ef7499f0d P a083da43d71847bda8d84f08dfb866be: Neither blocks nor log segments found. Creating new log.
I20260620 20:20:55.127140 4600 tablet_bootstrap.cc:492] T 57a20b63b778418d9e3d2b7ef7499f0d P 5ce0cbb9797a42399cee61c9291d59e1: No bootstrap required, opened a new log
I20260620 20:20:55.127432 4602 log.cc:824] T 57a20b63b778418d9e3d2b7ef7499f0d P a083da43d71847bda8d84f08dfb866be: Log is configured to *not* fsync() on all Append() calls
I20260620 20:20:55.127939 4600 ts_tablet_manager.cc:1397] T 57a20b63b778418d9e3d2b7ef7499f0d P 5ce0cbb9797a42399cee61c9291d59e1: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.005s
I20260620 20:20:55.128116 4601 tablet_bootstrap.cc:654] T 57a20b63b778418d9e3d2b7ef7499f0d P 73c7ac4da1144abeaac551bc31c42ec1: Neither blocks nor log segments found. Creating new log.
I20260620 20:20:55.129066 4601 log.cc:824] T 57a20b63b778418d9e3d2b7ef7499f0d P 73c7ac4da1144abeaac551bc31c42ec1: Log is configured to *not* fsync() on all Append() calls
I20260620 20:20:55.132226 4601 tablet_bootstrap.cc:492] T 57a20b63b778418d9e3d2b7ef7499f0d P 73c7ac4da1144abeaac551bc31c42ec1: No bootstrap required, opened a new log
I20260620 20:20:55.132483 4601 ts_tablet_manager.cc:1397] T 57a20b63b778418d9e3d2b7ef7499f0d P 73c7ac4da1144abeaac551bc31c42ec1: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260620 20:20:55.136535 4602 tablet_bootstrap.cc:492] T 57a20b63b778418d9e3d2b7ef7499f0d P a083da43d71847bda8d84f08dfb866be: No bootstrap required, opened a new log
I20260620 20:20:55.136821 4602 ts_tablet_manager.cc:1397] T 57a20b63b778418d9e3d2b7ef7499f0d P a083da43d71847bda8d84f08dfb866be: Time spent bootstrapping tablet: real 0.015s user 0.009s sys 0.003s
I20260620 20:20:55.140182 4601 raft_consensus.cc:348] T 57a20b63b778418d9e3d2b7ef7499f0d P 73c7ac4da1144abeaac551bc31c42ec1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73c7ac4da1144abeaac551bc31c42ec1" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42797 } } peers { permanent_uuid: "5ce0cbb9797a42399cee61c9291d59e1" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 41283 } } peers { permanent_uuid: "a083da43d71847bda8d84f08dfb866be" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 34483 } }
I20260620 20:20:55.140534 4601 raft_consensus.cc:374] T 57a20b63b778418d9e3d2b7ef7499f0d P 73c7ac4da1144abeaac551bc31c42ec1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:20:55.140671 4601 raft_consensus.cc:729] T 57a20b63b778418d9e3d2b7ef7499f0d P 73c7ac4da1144abeaac551bc31c42ec1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73c7ac4da1144abeaac551bc31c42ec1, State: Initialized, Role: FOLLOWER
I20260620 20:20:55.140616 4600 raft_consensus.cc:348] T 57a20b63b778418d9e3d2b7ef7499f0d P 5ce0cbb9797a42399cee61c9291d59e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73c7ac4da1144abeaac551bc31c42ec1" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42797 } } peers { permanent_uuid: "5ce0cbb9797a42399cee61c9291d59e1" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 41283 } } peers { permanent_uuid: "a083da43d71847bda8d84f08dfb866be" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 34483 } }
I20260620 20:20:55.141155 4600 raft_consensus.cc:374] T 57a20b63b778418d9e3d2b7ef7499f0d P 5ce0cbb9797a42399cee61c9291d59e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:20:55.141302 4600 raft_consensus.cc:729] T 57a20b63b778418d9e3d2b7ef7499f0d P 5ce0cbb9797a42399cee61c9291d59e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ce0cbb9797a42399cee61c9291d59e1, State: Initialized, Role: FOLLOWER
I20260620 20:20:55.141371 4601 consensus_queue.cc:260] T 57a20b63b778418d9e3d2b7ef7499f0d P 73c7ac4da1144abeaac551bc31c42ec1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73c7ac4da1144abeaac551bc31c42ec1" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42797 } } peers { permanent_uuid: "5ce0cbb9797a42399cee61c9291d59e1" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 41283 } } peers { permanent_uuid: "a083da43d71847bda8d84f08dfb866be" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 34483 } }
I20260620 20:20:55.141949 4600 consensus_queue.cc:260] T 57a20b63b778418d9e3d2b7ef7499f0d P 5ce0cbb9797a42399cee61c9291d59e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73c7ac4da1144abeaac551bc31c42ec1" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42797 } } peers { permanent_uuid: "5ce0cbb9797a42399cee61c9291d59e1" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 41283 } } peers { permanent_uuid: "a083da43d71847bda8d84f08dfb866be" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 34483 } }
I20260620 20:20:55.142624 4581 heartbeater.cc:503] Master 127.16.217.62:35921 was elected leader, sending a full tablet report...
I20260620 20:20:55.143321 4601 ts_tablet_manager.cc:1428] T 57a20b63b778418d9e3d2b7ef7499f0d P 73c7ac4da1144abeaac551bc31c42ec1: Time spent starting tablet: real 0.011s user 0.011s sys 0.000s
I20260620 20:20:55.149848 4602 raft_consensus.cc:348] T 57a20b63b778418d9e3d2b7ef7499f0d P a083da43d71847bda8d84f08dfb866be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73c7ac4da1144abeaac551bc31c42ec1" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42797 } } peers { permanent_uuid: "5ce0cbb9797a42399cee61c9291d59e1" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 41283 } } peers { permanent_uuid: "a083da43d71847bda8d84f08dfb866be" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 34483 } }
I20260620 20:20:55.150379 4600 ts_tablet_manager.cc:1428] T 57a20b63b778418d9e3d2b7ef7499f0d P 5ce0cbb9797a42399cee61c9291d59e1: Time spent starting tablet: real 0.022s user 0.018s sys 0.003s
I20260620 20:20:55.150413 4602 raft_consensus.cc:374] T 57a20b63b778418d9e3d2b7ef7499f0d P a083da43d71847bda8d84f08dfb866be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:20:55.150509 4602 raft_consensus.cc:729] T 57a20b63b778418d9e3d2b7ef7499f0d P a083da43d71847bda8d84f08dfb866be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a083da43d71847bda8d84f08dfb866be, State: Initialized, Role: FOLLOWER
I20260620 20:20:55.151314 4602 consensus_queue.cc:260] T 57a20b63b778418d9e3d2b7ef7499f0d P a083da43d71847bda8d84f08dfb866be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73c7ac4da1144abeaac551bc31c42ec1" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42797 } } peers { permanent_uuid: "5ce0cbb9797a42399cee61c9291d59e1" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 41283 } } peers { permanent_uuid: "a083da43d71847bda8d84f08dfb866be" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 34483 } }
I20260620 20:20:55.152558 4447 heartbeater.cc:503] Master 127.16.217.62:35921 was elected leader, sending a full tablet report...
I20260620 20:20:55.153280 4602 ts_tablet_manager.cc:1428] T 57a20b63b778418d9e3d2b7ef7499f0d P a083da43d71847bda8d84f08dfb866be: Time spent starting tablet: real 0.016s user 0.013s sys 0.004s
I20260620 20:20:55.174299 4607 raft_consensus.cc:482] T 57a20b63b778418d9e3d2b7ef7499f0d P 5ce0cbb9797a42399cee61c9291d59e1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 20:20:55.174633 4607 raft_consensus.cc:504] T 57a20b63b778418d9e3d2b7ef7499f0d P 5ce0cbb9797a42399cee61c9291d59e1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73c7ac4da1144abeaac551bc31c42ec1" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42797 } } peers { permanent_uuid: "5ce0cbb9797a42399cee61c9291d59e1" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 41283 } } peers { permanent_uuid: "a083da43d71847bda8d84f08dfb866be" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 34483 } }
I20260620 20:20:55.176360 4607 leader_election.cc:288] T 57a20b63b778418d9e3d2b7ef7499f0d P 5ce0cbb9797a42399cee61c9291d59e1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 73c7ac4da1144abeaac551bc31c42ec1 (127.16.217.3:42797), a083da43d71847bda8d84f08dfb866be (127.16.217.2:34483)
I20260620 20:20:55.183580 4598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:55.176812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.217.1:58029 (local address 127.16.217.2:34483)
0620 20:20:55.177013 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0620 20:20:55.177020 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:20:55.177580 (+ 560us) server_negotiation.cc:407] Connection header received
0620 20:20:55.177795 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:20:55.177802 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:20:55.177936 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:20:55.178058 (+ 122us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:20:55.178668 (+ 610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:55.179605 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:55.181474 (+ 1869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:55.182534 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:55.182659 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:20:55.182774 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0620 20:20:55.183119 (+ 345us) server_negotiation.cc:299] Negotiation successful
0620 20:20:55.183272 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 20:20:55.183652 4609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:55.176777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.2:34483 (local address 127.16.217.1:58029)
0620 20:20:55.177446 (+ 669us) negotiation.cc:107] Waiting for socket to connect
0620 20:20:55.177474 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0620 20:20:55.177643 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:20:55.178069 (+ 426us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:20:55.178072 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:20:55.178095 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:20:55.178505 (+ 410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:20:55.178517 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:20:55.179727 (+ 1210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:20:55.179733 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:20:55.181300 (+ 1567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:20:55.181312 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:20:55.182735 (+ 1423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:20:55.182743 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:20:55.182917 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:20:55.182955 (+ 38us) client_negotiation.cc:769] Sending connection context
0620 20:20:55.183080 (+ 125us) client_negotiation.cc:240] Negotiation successful
0620 20:20:55.183273 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":515,"thread_start_us":643,"threads_started":1}
I20260620 20:20:55.184087 4595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:55.176883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.217.1:54315 (local address 127.16.217.3:42797)
0620 20:20:55.177091 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0620 20:20:55.177095 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:20:55.178165 (+ 1070us) server_negotiation.cc:407] Connection header received
0620 20:20:55.178337 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:20:55.178342 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:20:55.178419 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:20:55.178521 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:20:55.179149 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:55.179938 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:55.182084 (+ 2146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:55.183009 (+ 925us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:55.183167 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:20:55.183233 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 20:20:55.183672 (+ 439us) server_negotiation.cc:299] Negotiation successful
0620 20:20:55.183828 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 20:20:55.184660 4610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:55.177186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.3:42797 (local address 127.16.217.1:54315)
0620 20:20:55.178053 (+ 867us) negotiation.cc:107] Waiting for socket to connect
0620 20:20:55.178077 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 20:20:55.178192 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:20:55.178569 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:20:55.178572 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:20:55.178619 (+ 47us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:20:55.178980 (+ 361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:20:55.178993 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:20:55.180096 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:20:55.180101 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:20:55.181832 (+ 1731us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:20:55.181900 (+ 68us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:20:55.183253 (+ 1353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:20:55.183259 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:20:55.183448 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:20:55.183493 (+ 45us) client_negotiation.cc:769] Sending connection context
0620 20:20:55.184282 (+ 789us) client_negotiation.cc:240] Negotiation successful
0620 20:20:55.184422 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":728,"thread_start_us":497,"threads_started":1}
I20260620 20:20:55.184779 4402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57a20b63b778418d9e3d2b7ef7499f0d" candidate_uuid: "5ce0cbb9797a42399cee61c9291d59e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a083da43d71847bda8d84f08dfb866be" is_pre_election: true
I20260620 20:20:55.185285 4402 raft_consensus.cc:2454] T 57a20b63b778418d9e3d2b7ef7499f0d P a083da43d71847bda8d84f08dfb866be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ce0cbb9797a42399cee61c9291d59e1 in term 0.
I20260620 20:20:55.185742 4536 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57a20b63b778418d9e3d2b7ef7499f0d" candidate_uuid: "5ce0cbb9797a42399cee61c9291d59e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c7ac4da1144abeaac551bc31c42ec1" is_pre_election: true
I20260620 20:20:55.185938 4203 leader_election.cc:302] T 57a20b63b778418d9e3d2b7ef7499f0d P 5ce0cbb9797a42399cee61c9291d59e1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ce0cbb9797a42399cee61c9291d59e1, a083da43d71847bda8d84f08dfb866be; no voters:
I20260620 20:20:55.186328 4607 raft_consensus.cc:2781] T 57a20b63b778418d9e3d2b7ef7499f0d P 5ce0cbb9797a42399cee61c9291d59e1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 20:20:55.186305 4536 raft_consensus.cc:2454] T 57a20b63b778418d9e3d2b7ef7499f0d P 73c7ac4da1144abeaac551bc31c42ec1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ce0cbb9797a42399cee61c9291d59e1 in term 0.
I20260620 20:20:55.186527 4607 raft_consensus.cc:482] T 57a20b63b778418d9e3d2b7ef7499f0d P 5ce0cbb9797a42399cee61c9291d59e1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 20:20:55.186671 4607 raft_consensus.cc:3037] T 57a20b63b778418d9e3d2b7ef7499f0d P 5ce0cbb9797a42399cee61c9291d59e1 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:20:55.189246 4607 raft_consensus.cc:504] T 57a20b63b778418d9e3d2b7ef7499f0d P 5ce0cbb9797a42399cee61c9291d59e1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73c7ac4da1144abeaac551bc31c42ec1" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42797 } } peers { permanent_uuid: "5ce0cbb9797a42399cee61c9291d59e1" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 41283 } } peers { permanent_uuid: "a083da43d71847bda8d84f08dfb866be" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 34483 } }
I20260620 20:20:55.189937 4607 leader_election.cc:288] T 57a20b63b778418d9e3d2b7ef7499f0d P 5ce0cbb9797a42399cee61c9291d59e1 [CANDIDATE]: Term 1 election: Requested vote from peers 73c7ac4da1144abeaac551bc31c42ec1 (127.16.217.3:42797), a083da43d71847bda8d84f08dfb866be (127.16.217.2:34483)
I20260620 20:20:55.190542 4402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57a20b63b778418d9e3d2b7ef7499f0d" candidate_uuid: "5ce0cbb9797a42399cee61c9291d59e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a083da43d71847bda8d84f08dfb866be"
I20260620 20:20:55.190797 4402 raft_consensus.cc:3037] T 57a20b63b778418d9e3d2b7ef7499f0d P a083da43d71847bda8d84f08dfb866be [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:20:55.193011 4402 raft_consensus.cc:2454] T 57a20b63b778418d9e3d2b7ef7499f0d P a083da43d71847bda8d84f08dfb866be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ce0cbb9797a42399cee61c9291d59e1 in term 1.
I20260620 20:20:55.193449 4203 leader_election.cc:302] T 57a20b63b778418d9e3d2b7ef7499f0d P 5ce0cbb9797a42399cee61c9291d59e1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ce0cbb9797a42399cee61c9291d59e1, a083da43d71847bda8d84f08dfb866be; no voters:
I20260620 20:20:55.193758 4607 raft_consensus.cc:2781] T 57a20b63b778418d9e3d2b7ef7499f0d P 5ce0cbb9797a42399cee61c9291d59e1 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:20:55.194281 4607 raft_consensus.cc:686] T 57a20b63b778418d9e3d2b7ef7499f0d P 5ce0cbb9797a42399cee61c9291d59e1 [term 1 LEADER]: Becoming Leader. State: Replica: 5ce0cbb9797a42399cee61c9291d59e1, State: Running, Role: LEADER
I20260620 20:20:55.194577 4536 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57a20b63b778418d9e3d2b7ef7499f0d" candidate_uuid: "5ce0cbb9797a42399cee61c9291d59e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c7ac4da1144abeaac551bc31c42ec1"
I20260620 20:20:55.194810 4536 raft_consensus.cc:3037] T 57a20b63b778418d9e3d2b7ef7499f0d P 73c7ac4da1144abeaac551bc31c42ec1 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:20:55.194857 4607 consensus_queue.cc:237] T 57a20b63b778418d9e3d2b7ef7499f0d P 5ce0cbb9797a42399cee61c9291d59e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73c7ac4da1144abeaac551bc31c42ec1" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42797 } } peers { permanent_uuid: "5ce0cbb9797a42399cee61c9291d59e1" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 41283 } } peers { permanent_uuid: "a083da43d71847bda8d84f08dfb866be" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 34483 } }
I20260620 20:20:55.198115 4536 raft_consensus.cc:2454] T 57a20b63b778418d9e3d2b7ef7499f0d P 73c7ac4da1144abeaac551bc31c42ec1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ce0cbb9797a42399cee61c9291d59e1 in term 1.
I20260620 20:20:55.201085 4124 catalog_manager.cc:5495] T 57a20b63b778418d9e3d2b7ef7499f0d P 5ce0cbb9797a42399cee61c9291d59e1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5ce0cbb9797a42399cee61c9291d59e1 (127.16.217.1). New cstate: current_term: 1 leader_uuid: "5ce0cbb9797a42399cee61c9291d59e1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73c7ac4da1144abeaac551bc31c42ec1" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42797 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5ce0cbb9797a42399cee61c9291d59e1" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 41283 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a083da43d71847bda8d84f08dfb866be" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 34483 } health_report { overall_health: UNKNOWN } } }
W20260620 20:20:55.214128 4314 tablet.cc:2369] T 57a20b63b778418d9e3d2b7ef7499f0d P 5ce0cbb9797a42399cee61c9291d59e1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 20:20:55.223287 4597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:55.217942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34996 (local address 127.16.217.1:41283)
0620 20:20:55.218144 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0620 20:20:55.218150 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:20:55.218169 (+ 19us) server_negotiation.cc:407] Connection header received
0620 20:20:55.218278 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:20:55.218283 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:20:55.218366 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:20:55.218528 (+ 162us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 20:20:55.218904 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:55.219831 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:55.221176 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:55.221580 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:55.221701 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:20:55.222210 (+ 509us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 20:20:55.222730 (+ 520us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 20:20:55.222856 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0620 20:20:55.222918 (+ 62us) server_negotiation.cc:299] Negotiation successful
0620 20:20:55.222983 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 20:20:55.231389 4402 raft_consensus.cc:1261] T 57a20b63b778418d9e3d2b7ef7499f0d P a083da43d71847bda8d84f08dfb866be [term 1 FOLLOWER]: Refusing update from remote peer 5ce0cbb9797a42399cee61c9291d59e1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:20:55.231389 4536 raft_consensus.cc:1261] T 57a20b63b778418d9e3d2b7ef7499f0d P 73c7ac4da1144abeaac551bc31c42ec1 [term 1 FOLLOWER]: Refusing update from remote peer 5ce0cbb9797a42399cee61c9291d59e1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260620 20:20:55.232295 4448 tablet.cc:2369] T 57a20b63b778418d9e3d2b7ef7499f0d P a083da43d71847bda8d84f08dfb866be: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 20:20:55.232414 4611 consensus_queue.cc:1035] T 57a20b63b778418d9e3d2b7ef7499f0d P 5ce0cbb9797a42399cee61c9291d59e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a083da43d71847bda8d84f08dfb866be" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 34483 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:20:55.232780 4607 consensus_queue.cc:1035] T 57a20b63b778418d9e3d2b7ef7499f0d P 5ce0cbb9797a42399cee61c9291d59e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "73c7ac4da1144abeaac551bc31c42ec1" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:20:55.258160 4124 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55008:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 20:20:55.258438 4124 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55008:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 20:20:55.261842 4124 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9ff3106ab4874c828568957739cb3a8b: Sending DeleteTablet for 3 replicas of tablet 57a20b63b778418d9e3d2b7ef7499f0d
I20260620 20:20:55.263046 4516 tablet_service.cc:1514] Processing DeleteTablet for tablet 57a20b63b778418d9e3d2b7ef7499f0d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:20:55 UTC) from {username='kudu', principal='kudu/127.16.217.62@KRBTEST.COM'} at 127.0.0.1:33336
I20260620 20:20:55.263070 4248 tablet_service.cc:1514] Processing DeleteTablet for tablet 57a20b63b778418d9e3d2b7ef7499f0d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:20:55 UTC) from {username='kudu', principal='kudu/127.16.217.62@KRBTEST.COM'} at 127.0.0.1:34994
I20260620 20:20:55.263365 4382 tablet_service.cc:1514] Processing DeleteTablet for tablet 57a20b63b778418d9e3d2b7ef7499f0d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:20:55 UTC) from {username='kudu', principal='kudu/127.16.217.62@KRBTEST.COM'} at 127.0.0.1:59218
I20260620 20:20:55.265300 17252 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu pid 4185
I20260620 20:20:55.281858 4624 tablet_replica.cc:330] T 57a20b63b778418d9e3d2b7ef7499f0d P 73c7ac4da1144abeaac551bc31c42ec1: stopping tablet replica
I20260620 20:20:55.282413 4626 tablet_replica.cc:330] T 57a20b63b778418d9e3d2b7ef7499f0d P a083da43d71847bda8d84f08dfb866be: stopping tablet replica
I20260620 20:20:55.283015 4626 raft_consensus.cc:2229] T 57a20b63b778418d9e3d2b7ef7499f0d P a083da43d71847bda8d84f08dfb866be [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 20:20:55.283015 4624 raft_consensus.cc:2229] T 57a20b63b778418d9e3d2b7ef7499f0d P 73c7ac4da1144abeaac551bc31c42ec1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 20:20:55.283545 4624 raft_consensus.cc:2258] T 57a20b63b778418d9e3d2b7ef7499f0d P 73c7ac4da1144abeaac551bc31c42ec1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:20:55.283562 4626 raft_consensus.cc:2258] T 57a20b63b778418d9e3d2b7ef7499f0d P a083da43d71847bda8d84f08dfb866be [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:20:55.286667 4626 ts_tablet_manager.cc:1905] T 57a20b63b778418d9e3d2b7ef7499f0d P a083da43d71847bda8d84f08dfb866be: Deleting tablet data with delete state TABLET_DATA_DELETED
Jun 20 20:20:55 dist-test-slave-2hpg krb5kdc[4079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986852, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.217.1@KRBTEST.COM
I20260620 20:20:55.287933 4624 ts_tablet_manager.cc:1905] T 57a20b63b778418d9e3d2b7ef7499f0d P 73c7ac4da1144abeaac551bc31c42ec1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:20:55.290772 4625 tablet_replica.cc:330] T 57a20b63b778418d9e3d2b7ef7499f0d P 5ce0cbb9797a42399cee61c9291d59e1: stopping tablet replica
I20260620 20:20:55.291460 4625 raft_consensus.cc:2229] T 57a20b63b778418d9e3d2b7ef7499f0d P 5ce0cbb9797a42399cee61c9291d59e1 [term 1 LEADER]: Raft consensus shutting down.
I20260620 20:20:55.292348 4625 raft_consensus.cc:2258] T 57a20b63b778418d9e3d2b7ef7499f0d P 5ce0cbb9797a42399cee61c9291d59e1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:20:55.294800 4626 ts_tablet_manager.cc:1918] T 57a20b63b778418d9e3d2b7ef7499f0d P a083da43d71847bda8d84f08dfb866be: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 20:20:55.295042 4626 log.cc:1196] T 57a20b63b778418d9e3d2b7ef7499f0d P a083da43d71847bda8d84f08dfb866be: Deleting WAL directory at /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986536687152-17252-0/minicluster-data/ts-1/wal/wals/57a20b63b778418d9e3d2b7ef7499f0d
I20260620 20:20:55.295702 4626 ts_tablet_manager.cc:1939] T 57a20b63b778418d9e3d2b7ef7499f0d P a083da43d71847bda8d84f08dfb866be: Deleting consensus metadata
I20260620 20:20:55.297672 4110 catalog_manager.cc:4841] TS a083da43d71847bda8d84f08dfb866be (127.16.217.2:34483): tablet 57a20b63b778418d9e3d2b7ef7499f0d (table test-table [id=4f413f8a26de449387d5163b25d85bbe]) successfully deleted
I20260620 20:20:55.297039 4597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:55.266637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35006 (local address 127.16.217.1:41283)
0620 20:20:55.266836 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0620 20:20:55.266844 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 20:20:55.272020 (+ 5176us) server_negotiation.cc:407] Connection header received
0620 20:20:55.272982 (+ 962us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:20:55.272991 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:20:55.273105 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:20:55.273198 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:20:55.277603 (+ 4405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:55.279250 (+ 1647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:55.282327 (+ 3077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:55.283060 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:55.283174 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:20:55.288850 (+ 5676us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:20:55.288865 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:20:55.288883 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:20:55.288932 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:20:55.293279 (+ 4347us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:20:55.294280 (+ 1001us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:20:55.294291 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:20:55.294295 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:20:55.294478 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:20:55.294970 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:20:55.294976 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:20:55.294979 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:20:55.295418 (+ 439us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:20:55.295727 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0620 20:20:55.296361 (+ 634us) server_negotiation.cc:299] Negotiation successful
0620 20:20:55.296655 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 20:20:55.298570 4308 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:20:55.299567 4624 ts_tablet_manager.cc:1918] T 57a20b63b778418d9e3d2b7ef7499f0d P 73c7ac4da1144abeaac551bc31c42ec1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 20:20:55.299770 4624 log.cc:1196] T 57a20b63b778418d9e3d2b7ef7499f0d P 73c7ac4da1144abeaac551bc31c42ec1: Deleting WAL directory at /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986536687152-17252-0/minicluster-data/ts-2/wal/wals/57a20b63b778418d9e3d2b7ef7499f0d
I20260620 20:20:55.300470 4624 ts_tablet_manager.cc:1939] T 57a20b63b778418d9e3d2b7ef7499f0d P 73c7ac4da1144abeaac551bc31c42ec1: Deleting consensus metadata
I20260620 20:20:55.302698 4111 catalog_manager.cc:4841] TS 73c7ac4da1144abeaac551bc31c42ec1 (127.16.217.3:42797): tablet 57a20b63b778418d9e3d2b7ef7499f0d (table test-table [id=4f413f8a26de449387d5163b25d85bbe]) successfully deleted
I20260620 20:20:55.397171 17252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu with pid 4185
I20260620 20:20:55.415035 17252 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu pid 4319
W20260620 20:20:55.414950 4109 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.16.217.1:41283 (error 108)
I20260620 20:20:55.416805 4599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:55.416412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.1:41283 (local address 127.0.0.1:35020)
0620 20:20:55.416570 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0620 20:20:55.416670 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.217.1:41283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260620 20:20:55.417047 4109 catalog_manager.cc:4584] TS 5ce0cbb9797a42399cee61c9291d59e1 (127.16.217.1:41283): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 57a20b63b778418d9e3d2b7ef7499f0d: Network error: Client connection negotiation failed: client connection to 127.16.217.1:41283: connect: Connection refused (error 111)
Jun 20 20:20:55 dist-test-slave-2hpg krb5kdc[4079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986852, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.217.2@KRBTEST.COM
I20260620 20:20:55.429736 4598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:55.415771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59228 (local address 127.16.217.2:34483)
0620 20:20:55.415953 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0620 20:20:55.415959 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:20:55.416094 (+ 135us) server_negotiation.cc:407] Connection header received
0620 20:20:55.416297 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:20:55.416301 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:20:55.416364 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:20:55.416482 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:20:55.417826 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:55.418682 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:55.420018 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:55.420386 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:55.420513 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:20:55.424568 (+ 4055us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:20:55.424581 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:20:55.424596 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:20:55.424631 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:20:55.427389 (+ 2758us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:20:55.427998 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:20:55.428005 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:20:55.428007 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:20:55.428116 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:20:55.428413 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:20:55.428416 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:20:55.428419 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:20:55.428764 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:20:55.428972 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0620 20:20:55.429217 (+ 245us) server_negotiation.cc:299] Negotiation successful
0620 20:20:55.429370 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 20:20:55.430608 4442 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:20:55.470748 4599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:55.470222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.1:41283 (local address 127.0.0.1:35028)
0620 20:20:55.470449 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0620 20:20:55.470593 (+ 144us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.217.1:41283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260620 20:20:55.471798 4599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:55.471438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.1:41283 (local address 127.0.0.1:35042)
0620 20:20:55.471596 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0620 20:20:55.471674 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.217.1:41283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260620 20:20:55.519677 17252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu with pid 4319
I20260620 20:20:55.532621 4599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:55.531965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.1:41283 (local address 127.0.0.1:35048)
0620 20:20:55.532262 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0620 20:20:55.532389 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.217.1:41283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20260620 20:20:55.533843 4599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:55.533390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.1:41283 (local address 127.0.0.1:35062)
0620 20:20:55.533573 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0620 20:20:55.533676 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.217.1:41283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260620 20:20:55.537070 17252 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu pid 4453
Jun 20 20:20:55 dist-test-slave-2hpg krb5kdc[4079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986852, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.217.3@KRBTEST.COM
I20260620 20:20:55.552673 4595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:55.537782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33352 (local address 127.16.217.3:42797)
0620 20:20:55.538015 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0620 20:20:55.538021 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:20:55.538042 (+ 21us) server_negotiation.cc:407] Connection header received
0620 20:20:55.538211 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:20:55.538217 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:20:55.538293 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:20:55.538398 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:20:55.539767 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:55.540509 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:55.541847 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:55.542333 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:55.542469 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:20:55.547021 (+ 4552us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:20:55.547053 (+ 32us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:20:55.547070 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:20:55.547108 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:20:55.549724 (+ 2616us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:20:55.550484 (+ 760us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:20:55.550494 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:20:55.550497 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:20:55.550683 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:20:55.551086 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:20:55.551091 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:20:55.551094 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:20:55.551440 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:20:55.551772 (+ 332us) server_negotiation.cc:1035] Waiting for connection context
0620 20:20:55.552065 (+ 293us) server_negotiation.cc:299] Negotiation successful
0620 20:20:55.552291 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260620 20:20:55.553408 4576 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:20:55.614537 4599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:55.614072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.1:41283 (local address 127.0.0.1:35078)
0620 20:20:55.614283 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0620 20:20:55.614378 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.217.1:41283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260620 20:20:55.615711 4599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:55.615309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.1:41283 (local address 127.0.0.1:35090)
0620 20:20:55.615525 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0620 20:20:55.615601 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.217.1:41283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112}
I20260620 20:20:55.643093 17252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu with pid 4453
I20260620 20:20:55.659890 17252 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu pid 4092
I20260620 20:20:55.661024 4154 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:20:55.770021 17252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu with pid 4092
2026-06-20T20:20:55Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (4221 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-06-20T20:20:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T20:20:55Z Disabled control of system clock
I20260620 20:20:55.846482 17252 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
/tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986536687152-17252-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986536687152-17252-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986536687152-17252-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986536687152-17252-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.217.62:46265
--webserver_interface=127.16.217.62
--webserver_port=0
--builtin_ntp_servers=127.16.217.20:46149
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.217.62:46265
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 20:20:56.204094 4636 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:20:56.204414 4636 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:20:56.204576 4636 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:20:56.212191 4636 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:20:56.212311 4636 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:20:56.212396 4636 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:20:56.212527 4636 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:20:56.212615 4636 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:20:56.222536 4636 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.217.20:46149
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986536687152-17252-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986536687152-17252-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.217.62:46265
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.217.62:46265
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986536687152-17252-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.217.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986536687152-17252-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:20:56.223857 4636 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:20:56.225651 4636 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:20:56.237553 4644 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:20:56.237602 4642 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:20:56.238090 4641 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:20:56.238412 4636 server_base.cc:1034] running on GCE node
I20260620 20:20:56.239234 4636 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:20:56.240651 4636 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:20:56.241850 4636 hybrid_clock.cc:648] HybridClock initialized: now 1781986856241787 us; error 62 us; skew 500 ppm
I20260620 20:20:56.242262 4636 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 20:20:56.244493 4636 webserver.cc:460] Webserver started at http://127.16.217.62:45753/ using document root <none> and password file <none>
I20260620 20:20:56.245189 4636 fs_manager.cc:359] Metadata directory not provided
I20260620 20:20:56.245301 4636 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:20:56.245637 4636 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:20:56.248574 4636 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986536687152-17252-0/minicluster-data/master-0/data/instance:
uuid: "e76f35a6e1134497b75e69e38296cc7c"
format_stamp: "Formatted at 2026-06-20 20:20:56 on dist-test-slave-2hpg"
server_key: "216afeef92a70d4eea82f8bcf84b2ca8"
server_key_iv: "7833696ad9ce768189eb97bbadcd9356"
server_key_version: "encryptionkey@0"
I20260620 20:20:56.249418 4636 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986536687152-17252-0/minicluster-data/master-0/wal/instance:
uuid: "e76f35a6e1134497b75e69e38296cc7c"
format_stamp: "Formatted at 2026-06-20 20:20:56 on dist-test-slave-2hpg"
server_key: "216afeef92a70d4eea82f8bcf84b2ca8"
server_key_iv: "7833696ad9ce768189eb97bbadcd9356"
server_key_version: "encryptionkey@0"
I20260620 20:20:56.254475 4636 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260620 20:20:56.257829 4650 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:20:56.259194 4636 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 20:20:56.259382 4636 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986536687152-17252-0/minicluster-data/master-0/data,/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986536687152-17252-0/minicluster-data/master-0/wal
uuid: "e76f35a6e1134497b75e69e38296cc7c"
format_stamp: "Formatted at 2026-06-20 20:20:56 on dist-test-slave-2hpg"
server_key: "216afeef92a70d4eea82f8bcf84b2ca8"
server_key_iv: "7833696ad9ce768189eb97bbadcd9356"
server_key_version: "encryptionkey@0"
I20260620 20:20:56.259611 4636 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986536687152-17252-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986536687152-17252-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986536687152-17252-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:20:56.275051 4636 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:20:56.275978 4636 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:20:56.276302 4636 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:20:56.295987 4636 rpc_server.cc:307] RPC server started. Bound to: 127.16.217.62:46265
I20260620 20:20:56.296025 4701 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.217.62:46265 every 8 connection(s)
I20260620 20:20:56.297402 4636 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986536687152-17252-0/minicluster-data/master-0/data/info.pb
I20260620 20:20:56.298543 17252 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu as pid 4636
I20260620 20:20:56.298739 17252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986536687152-17252-0/minicluster-data/master-0/wal/instance
I20260620 20:20:56.299312 17252 external_mini_cluster.cc:1428] Setting key 0b40d4c5b88d2764c0a8d296d2610682
I20260620 20:20:56.302464 4702 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:20:56.313231 4705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:56.301488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44382 (local address 127.16.217.62:46265)
0620 20:20:56.302508 (+ 1020us) server_negotiation.cc:206] Beginning negotiation
0620 20:20:56.302544 (+ 36us) server_negotiation.cc:399] Waiting for connection header
0620 20:20:56.302607 (+ 63us) server_negotiation.cc:407] Connection header received
0620 20:20:56.304254 (+ 1647us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:20:56.304276 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:20:56.304710 (+ 434us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:20:56.305101 (+ 391us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:20:56.305679 (+ 578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:56.307027 (+ 1348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:56.308381 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:56.309077 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:56.309286 (+ 209us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:20:56.310687 (+ 1401us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:20:56.310695 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:20:56.310712 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:20:56.311366 (+ 654us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:20:56.311445 (+ 79us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:20:56.311470 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:20:56.311652 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0620 20:20:56.311889 (+ 237us) server_negotiation.cc:299] Negotiation successful
0620 20:20:56.312258 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":605,"thread_start_us":404,"threads_started":1}
I20260620 20:20:56.314597 4702 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e76f35a6e1134497b75e69e38296cc7c: Bootstrap starting.
I20260620 20:20:56.318269 4702 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e76f35a6e1134497b75e69e38296cc7c: Neither blocks nor log segments found. Creating new log.
I20260620 20:20:56.319413 4702 log.cc:824] T 00000000000000000000000000000000 P e76f35a6e1134497b75e69e38296cc7c: Log is configured to *not* fsync() on all Append() calls
I20260620 20:20:56.322770 4702 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e76f35a6e1134497b75e69e38296cc7c: No bootstrap required, opened a new log
I20260620 20:20:56.330289 4702 raft_consensus.cc:348] T 00000000000000000000000000000000 P e76f35a6e1134497b75e69e38296cc7c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e76f35a6e1134497b75e69e38296cc7c" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 46265 } }
I20260620 20:20:56.330698 4702 raft_consensus.cc:374] T 00000000000000000000000000000000 P e76f35a6e1134497b75e69e38296cc7c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:20:56.330821 4702 raft_consensus.cc:729] T 00000000000000000000000000000000 P e76f35a6e1134497b75e69e38296cc7c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e76f35a6e1134497b75e69e38296cc7c, State: Initialized, Role: FOLLOWER
I20260620 20:20:56.331403 4702 consensus_queue.cc:260] T 00000000000000000000000000000000 P e76f35a6e1134497b75e69e38296cc7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e76f35a6e1134497b75e69e38296cc7c" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 46265 } }
I20260620 20:20:56.331650 4702 raft_consensus.cc:388] T 00000000000000000000000000000000 P e76f35a6e1134497b75e69e38296cc7c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 20:20:56.331789 4702 raft_consensus.cc:482] T 00000000000000000000000000000000 P e76f35a6e1134497b75e69e38296cc7c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 20:20:56.331975 4702 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e76f35a6e1134497b75e69e38296cc7c [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:20:56.334213 4702 raft_consensus.cc:504] T 00000000000000000000000000000000 P e76f35a6e1134497b75e69e38296cc7c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e76f35a6e1134497b75e69e38296cc7c" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 46265 } }
I20260620 20:20:56.334748 4702 leader_election.cc:302] T 00000000000000000000000000000000 P e76f35a6e1134497b75e69e38296cc7c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e76f35a6e1134497b75e69e38296cc7c; no voters:
I20260620 20:20:56.335345 4702 leader_election.cc:288] T 00000000000000000000000000000000 P e76f35a6e1134497b75e69e38296cc7c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 20:20:56.335515 4707 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e76f35a6e1134497b75e69e38296cc7c [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:20:56.336232 4707 raft_consensus.cc:686] T 00000000000000000000000000000000 P e76f35a6e1134497b75e69e38296cc7c [term 1 LEADER]: Becoming Leader. State: Replica: e76f35a6e1134497b75e69e38296cc7c, State: Running, Role: LEADER
I20260620 20:20:56.336825 4707 consensus_queue.cc:237] T 00000000000000000000000000000000 P e76f35a6e1134497b75e69e38296cc7c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e76f35a6e1134497b75e69e38296cc7c" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 46265 } }
I20260620 20:20:56.337560 4702 sys_catalog.cc:564] T 00000000000000000000000000000000 P e76f35a6e1134497b75e69e38296cc7c [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:20:56.340134 4708 sys_catalog.cc:455] T 00000000000000000000000000000000 P e76f35a6e1134497b75e69e38296cc7c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e76f35a6e1134497b75e69e38296cc7c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e76f35a6e1134497b75e69e38296cc7c" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 46265 } } }
I20260620 20:20:56.340159 4709 sys_catalog.cc:455] T 00000000000000000000000000000000 P e76f35a6e1134497b75e69e38296cc7c [sys.catalog]: SysCatalogTable state changed. Reason: New leader e76f35a6e1134497b75e69e38296cc7c. Latest consensus state: current_term: 1 leader_uuid: "e76f35a6e1134497b75e69e38296cc7c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e76f35a6e1134497b75e69e38296cc7c" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 46265 } } }
I20260620 20:20:56.340644 4708 sys_catalog.cc:458] T 00000000000000000000000000000000 P e76f35a6e1134497b75e69e38296cc7c [sys.catalog]: This master's current role is: LEADER
I20260620 20:20:56.340642 4709 sys_catalog.cc:458] T 00000000000000000000000000000000 P e76f35a6e1134497b75e69e38296cc7c [sys.catalog]: This master's current role is: LEADER
I20260620 20:20:56.347349 4716 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 20:20:56.353029 4716 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 20:20:56.361977 4716 catalog_manager.cc:1347] Generated new cluster ID: 38745277aaf0401a91a23fcd6e645044
I20260620 20:20:56.362115 4716 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 20:20:56.375402 4716 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 20:20:56.377231 4716 catalog_manager.cc:1504] Loading token signing keys...
I20260620 20:20:56.393901 4716 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e76f35a6e1134497b75e69e38296cc7c: Generated new TSK 0
I20260620 20:20:56.394798 4716 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 20:20:56.411814 17252 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
/tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986536687152-17252-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986536687152-17252-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986536687152-17252-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986536687152-17252-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.217.1:0
--local_ip_for_outbound_sockets=127.16.217.1
--webserver_interface=127.16.217.1
--webserver_port=0
--tserver_master_addrs=127.16.217.62:46265
--builtin_ntp_servers=127.16.217.20:46149
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 20:20:56.741520 4726 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:20:56.741784 4726 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:20:56.741962 4726 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:20:56.749295 4726 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:20:56.749392 4726 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:20:56.749567 4726 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.217.1
I20260620 20:20:56.758884 4726 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.217.20:46149
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986536687152-17252-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986536687152-17252-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.217.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986536687152-17252-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.217.1
--webserver_port=0
--tserver_master_addrs=127.16.217.62:46265
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.217.1
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986536687152-17252-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:20:56.760098 4726 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:20:56.761969 4726 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:20:56.775282 4731 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:20:56.775241 4734 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:20:56.775327 4732 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:20:56.776134 4726 server_base.cc:1034] running on GCE node
I20260620 20:20:56.776750 4726 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:20:56.777683 4726 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:20:56.778918 4726 hybrid_clock.cc:648] HybridClock initialized: now 1781986856778870 us; error 58 us; skew 500 ppm
I20260620 20:20:56.779241 4726 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 20:20:56.784963 4726 webserver.cc:460] Webserver started at http://127.16.217.1:46467/ using document root <none> and password file <none>
I20260620 20:20:56.785673 4726 fs_manager.cc:359] Metadata directory not provided
I20260620 20:20:56.785806 4726 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:20:56.786160 4726 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:20:56.789026 4726 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986536687152-17252-0/minicluster-data/ts-0/data/instance:
uuid: "1c9e1103c5b14ed08970a95305274dc8"
format_stamp: "Formatted at 2026-06-20 20:20:56 on dist-test-slave-2hpg"
server_key: "878f404c15407690c4d307e7dc97c20d"
server_key_iv: "d6f26782ba64f9ceef20ea82f64fe200"
server_key_version: "encryptionkey@0"
I20260620 20:20:56.789829 4726 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986536687152-17252-0/minicluster-data/ts-0/wal/instance:
uuid: "1c9e1103c5b14ed08970a95305274dc8"
format_stamp: "Formatted at 2026-06-20 20:20:56 on dist-test-slave-2hpg"
server_key: "878f404c15407690c4d307e7dc97c20d"
server_key_iv: "d6f26782ba64f9ceef20ea82f64fe200"
server_key_version: "encryptionkey@0"
I20260620 20:20:56.794998 4726 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 20:20:56.798692 4740 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:20:56.800010 4726 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 20:20:56.800184 4726 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986536687152-17252-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986536687152-17252-0/minicluster-data/ts-0/wal
uuid: "1c9e1103c5b14ed08970a95305274dc8"
format_stamp: "Formatted at 2026-06-20 20:20:56 on dist-test-slave-2hpg"
server_key: "878f404c15407690c4d307e7dc97c20d"
server_key_iv: "d6f26782ba64f9ceef20ea82f64fe200"
server_key_version: "encryptionkey@0"
I20260620 20:20:56.800400 4726 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986536687152-17252-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986536687152-17252-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986536687152-17252-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:20:56.814049 4726 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:20:56.814845 4726 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:20:56.815151 4726 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:20:56.816089 4726 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:20:56.817481 4726 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:20:56.817615 4726 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:20:56.817749 4726 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:20:56.817837 4726 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:20:56.851918 4726 rpc_server.cc:307] RPC server started. Bound to: 127.16.217.1:34185
I20260620 20:20:56.852005 4852 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.217.1:34185 every 8 connection(s)
I20260620 20:20:56.853253 4726 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986536687152-17252-0/minicluster-data/ts-0/data/info.pb
I20260620 20:20:56.856297 17252 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu as pid 4726
I20260620 20:20:56.856470 17252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986536687152-17252-0/minicluster-data/ts-0/wal/instance
I20260620 20:20:56.856906 17252 external_mini_cluster.cc:1428] Setting key ada56a663f6a5cbaeef92dcdf6bde827
I20260620 20:20:56.859599 17252 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
/tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986536687152-17252-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986536687152-17252-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986536687152-17252-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986536687152-17252-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.217.2:0
--local_ip_for_outbound_sockets=127.16.217.2
--webserver_interface=127.16.217.2
--webserver_port=0
--tserver_master_addrs=127.16.217.62:46265
--builtin_ntp_servers=127.16.217.20:46149
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 20:20:56.867651 4855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:56.856068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.217.1:54813 (local address 127.16.217.62:46265)
0620 20:20:56.856591 (+ 523us) server_negotiation.cc:206] Beginning negotiation
0620 20:20:56.856597 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:20:56.857924 (+ 1327us) server_negotiation.cc:407] Connection header received
0620 20:20:56.859704 (+ 1780us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:20:56.859709 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:20:56.859794 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:20:56.859903 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:20:56.861440 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:56.862241 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:56.864702 (+ 2461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:56.865090 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:56.865186 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:20:56.866829 (+ 1643us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:20:56.866835 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:20:56.866838 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:20:56.866855 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:20:56.866894 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:20:56.866906 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:20:56.866977 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 20:20:56.867261 (+ 284us) server_negotiation.cc:299] Negotiation successful
0620 20:20:56.867340 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":375,"thread_start_us":245,"threads_started":1}
I20260620 20:20:56.868981 4856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:56.856710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:46265 (local address 127.16.217.1:54813)
0620 20:20:56.857740 (+ 1030us) negotiation.cc:107] Waiting for socket to connect
0620 20:20:56.857794 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0620 20:20:56.859447 (+ 1653us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:20:56.860161 (+ 714us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:20:56.860176 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:20:56.860297 (+ 121us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:20:56.861207 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:20:56.861252 (+ 45us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:20:56.862387 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:20:56.862395 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:20:56.864563 (+ 2168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:20:56.864581 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:20:56.865315 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:20:56.865325 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:20:56.865683 (+ 358us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:20:56.866469 (+ 786us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 20:20:56.866475 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:20:56.866528 (+ 53us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 20:20:56.866638 (+ 110us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:20:56.867005 (+ 367us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:20:56.867019 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:20:56.867030 (+ 11us) client_negotiation.cc:769] Sending connection context
0620 20:20:56.867213 (+ 183us) client_negotiation.cc:240] Negotiation successful
0620 20:20:56.867662 (+ 449us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":517,"thread_start_us":345,"threads_started":1}
I20260620 20:20:56.870625 4853 heartbeater.cc:347] Connected to a master server at 127.16.217.62:46265
I20260620 20:20:56.870929 4853 heartbeater.cc:464] Registering TS with master...
I20260620 20:20:56.871651 4853 heartbeater.cc:511] Master 127.16.217.62:46265 requested a full tablet report, sending...
I20260620 20:20:56.873631 4667 ts_manager.cc:194] Registered new tserver with Master: 1c9e1103c5b14ed08970a95305274dc8 (127.16.217.1:34185)
I20260620 20:20:56.875175 4667 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.217.1:54813
W20260620 20:20:57.187583 4857 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:20:57.187837 4857 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:20:57.187999 4857 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:20:57.197607 4857 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:20:57.197698 4857 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:20:57.197933 4857 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.217.2
I20260620 20:20:57.207250 4857 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.217.20:46149
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986536687152-17252-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986536687152-17252-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.217.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986536687152-17252-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.217.2
--webserver_port=0
--tserver_master_addrs=127.16.217.62:46265
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.217.2
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986536687152-17252-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:20:57.208525 4857 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:20:57.210366 4857 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:20:57.223402 4865 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:20:57.223498 4862 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:20:57.223439 4863 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:20:57.224360 4857 server_base.cc:1034] running on GCE node
I20260620 20:20:57.224977 4857 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:20:57.225970 4857 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:20:57.227175 4857 hybrid_clock.cc:648] HybridClock initialized: now 1781986857227132 us; error 36 us; skew 500 ppm
I20260620 20:20:57.227495 4857 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 20:20:57.229669 4857 webserver.cc:460] Webserver started at http://127.16.217.2:42287/ using document root <none> and password file <none>
I20260620 20:20:57.230410 4857 fs_manager.cc:359] Metadata directory not provided
I20260620 20:20:57.230522 4857 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:20:57.230839 4857 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:20:57.233647 4857 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986536687152-17252-0/minicluster-data/ts-1/data/instance:
uuid: "a75146fcda6c47328e38eb51dbb35a07"
format_stamp: "Formatted at 2026-06-20 20:20:57 on dist-test-slave-2hpg"
server_key: "d1f5eab5f9e8c58fe1de157939b7b11e"
server_key_iv: "b23820686284778647e2bec4e5affeba"
server_key_version: "encryptionkey@0"
I20260620 20:20:57.234553 4857 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986536687152-17252-0/minicluster-data/ts-1/wal/instance:
uuid: "a75146fcda6c47328e38eb51dbb35a07"
format_stamp: "Formatted at 2026-06-20 20:20:57 on dist-test-slave-2hpg"
server_key: "d1f5eab5f9e8c58fe1de157939b7b11e"
server_key_iv: "b23820686284778647e2bec4e5affeba"
server_key_version: "encryptionkey@0"
I20260620 20:20:57.239491 4857 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 20:20:57.242753 4871 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:20:57.244011 4857 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 20:20:57.244187 4857 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986536687152-17252-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986536687152-17252-0/minicluster-data/ts-1/wal
uuid: "a75146fcda6c47328e38eb51dbb35a07"
format_stamp: "Formatted at 2026-06-20 20:20:57 on dist-test-slave-2hpg"
server_key: "d1f5eab5f9e8c58fe1de157939b7b11e"
server_key_iv: "b23820686284778647e2bec4e5affeba"
server_key_version: "encryptionkey@0"
I20260620 20:20:57.244395 4857 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986536687152-17252-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986536687152-17252-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986536687152-17252-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:20:57.265200 4857 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:20:57.266141 4857 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:20:57.266446 4857 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:20:57.267402 4857 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:20:57.268863 4857 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:20:57.268985 4857 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:20:57.269120 4857 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:20:57.269207 4857 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:20:57.303231 4857 rpc_server.cc:307] RPC server started. Bound to: 127.16.217.2:46037
I20260620 20:20:57.303301 4983 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.217.2:46037 every 8 connection(s)
I20260620 20:20:57.304667 4857 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986536687152-17252-0/minicluster-data/ts-1/data/info.pb
I20260620 20:20:57.306159 17252 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu as pid 4857
I20260620 20:20:57.306350 17252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986536687152-17252-0/minicluster-data/ts-1/wal/instance
I20260620 20:20:57.306828 17252 external_mini_cluster.cc:1428] Setting key fbdfc09fd3c2efa5cbf43f53139d9b34
I20260620 20:20:57.309990 17252 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
/tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986536687152-17252-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986536687152-17252-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986536687152-17252-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986536687152-17252-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.217.3:0
--local_ip_for_outbound_sockets=127.16.217.3
--webserver_interface=127.16.217.3
--webserver_port=0
--tserver_master_addrs=127.16.217.62:46265
--builtin_ntp_servers=127.16.217.20:46149
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 20:20:57.319932 4855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:57.307591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.217.2:58823 (local address 127.16.217.62:46265)
0620 20:20:57.308554 (+ 963us) server_negotiation.cc:206] Beginning negotiation
0620 20:20:57.308560 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:20:57.309353 (+ 793us) server_negotiation.cc:407] Connection header received
0620 20:20:57.311949 (+ 2596us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:20:57.311957 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:20:57.312046 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:20:57.312164 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:20:57.313513 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:57.314702 (+ 1189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:57.316679 (+ 1977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:57.317593 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:57.317727 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:20:57.319251 (+ 1524us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:20:57.319256 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:20:57.319259 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:20:57.319276 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:20:57.319311 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:20:57.319322 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:20:57.319395 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 20:20:57.319642 (+ 247us) server_negotiation.cc:299] Negotiation successful
0620 20:20:57.319720 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":827}
I20260620 20:20:57.321421 4986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:57.308113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:46265 (local address 127.16.217.2:58823)
0620 20:20:57.309194 (+ 1081us) negotiation.cc:107] Waiting for socket to connect
0620 20:20:57.309252 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0620 20:20:57.311634 (+ 2382us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:20:57.312406 (+ 772us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:20:57.312419 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:20:57.312539 (+ 120us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:20:57.313331 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:20:57.313354 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:20:57.314895 (+ 1541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:20:57.314900 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:20:57.316517 (+ 1617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:20:57.316529 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:20:57.317824 (+ 1295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:20:57.317832 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:20:57.318108 (+ 276us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:20:57.318889 (+ 781us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 20:20:57.318894 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:20:57.318948 (+ 54us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 20:20:57.319051 (+ 103us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:20:57.319418 (+ 367us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:20:57.319431 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:20:57.319442 (+ 11us) client_negotiation.cc:769] Sending connection context
0620 20:20:57.319622 (+ 180us) client_negotiation.cc:240] Negotiation successful
0620 20:20:57.320052 (+ 430us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":519,"thread_start_us":351,"threads_started":1}
I20260620 20:20:57.322893 4984 heartbeater.cc:347] Connected to a master server at 127.16.217.62:46265
I20260620 20:20:57.323196 4984 heartbeater.cc:464] Registering TS with master...
I20260620 20:20:57.323935 4984 heartbeater.cc:511] Master 127.16.217.62:46265 requested a full tablet report, sending...
I20260620 20:20:57.325359 4667 ts_manager.cc:194] Registered new tserver with Master: a75146fcda6c47328e38eb51dbb35a07 (127.16.217.2:46037)
I20260620 20:20:57.326241 4667 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.217.2:58823
W20260620 20:20:57.640379 4987 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:20:57.640681 4987 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:20:57.640836 4987 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:20:57.648198 4987 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:20:57.648321 4987 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:20:57.648488 4987 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.217.3
I20260620 20:20:57.657490 4987 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.217.20:46149
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986536687152-17252-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986536687152-17252-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.217.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986536687152-17252-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.217.3
--webserver_port=0
--tserver_master_addrs=127.16.217.62:46265
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.217.3
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986536687152-17252-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:20:57.658798 4987 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:20:57.660531 4987 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:20:57.673550 4993 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:20:57.673600 4995 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:20:57.673624 4992 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:20:57.674379 4987 server_base.cc:1034] running on GCE node
I20260620 20:20:57.674897 4987 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:20:57.675745 4987 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:20:57.676985 4987 hybrid_clock.cc:648] HybridClock initialized: now 1781986857676937 us; error 54 us; skew 500 ppm
I20260620 20:20:57.677325 4987 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 20:20:57.679594 4987 webserver.cc:460] Webserver started at http://127.16.217.3:40275/ using document root <none> and password file <none>
I20260620 20:20:57.680269 4987 fs_manager.cc:359] Metadata directory not provided
I20260620 20:20:57.680403 4987 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:20:57.680722 4987 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:20:57.683539 4987 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986536687152-17252-0/minicluster-data/ts-2/data/instance:
uuid: "d490a6234f3e41bea8127b1de701d889"
format_stamp: "Formatted at 2026-06-20 20:20:57 on dist-test-slave-2hpg"
server_key: "f716e07aab9980c260159c4b5cce048f"
server_key_iv: "6b236aaec5d5cdd40f98b69a6fd2d53e"
server_key_version: "encryptionkey@0"
I20260620 20:20:57.684299 4987 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986536687152-17252-0/minicluster-data/ts-2/wal/instance:
uuid: "d490a6234f3e41bea8127b1de701d889"
format_stamp: "Formatted at 2026-06-20 20:20:57 on dist-test-slave-2hpg"
server_key: "f716e07aab9980c260159c4b5cce048f"
server_key_iv: "6b236aaec5d5cdd40f98b69a6fd2d53e"
server_key_version: "encryptionkey@0"
I20260620 20:20:57.689075 4987 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260620 20:20:57.692366 5001 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:20:57.693632 4987 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 20:20:57.693800 4987 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986536687152-17252-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986536687152-17252-0/minicluster-data/ts-2/wal
uuid: "d490a6234f3e41bea8127b1de701d889"
format_stamp: "Formatted at 2026-06-20 20:20:57 on dist-test-slave-2hpg"
server_key: "f716e07aab9980c260159c4b5cce048f"
server_key_iv: "6b236aaec5d5cdd40f98b69a6fd2d53e"
server_key_version: "encryptionkey@0"
I20260620 20:20:57.694080 4987 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986536687152-17252-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986536687152-17252-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986536687152-17252-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:20:57.720541 4987 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:20:57.721369 4987 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:20:57.721670 4987 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:20:57.722679 4987 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:20:57.724105 4987 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:20:57.724233 4987 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:20:57.724400 4987 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:20:57.724490 4987 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:20:57.757686 4987 rpc_server.cc:307] RPC server started. Bound to: 127.16.217.3:44509
I20260620 20:20:57.757776 5113 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.217.3:44509 every 8 connection(s)
I20260620 20:20:57.759086 4987 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986536687152-17252-0/minicluster-data/ts-2/data/info.pb
I20260620 20:20:57.766782 17252 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu as pid 4987
I20260620 20:20:57.766961 17252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986536687152-17252-0/minicluster-data/ts-2/wal/instance
I20260620 20:20:57.767379 17252 external_mini_cluster.cc:1428] Setting key dd3cca5081b3aae84a3fb66176e42ea5
I20260620 20:20:57.771721 4855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:57.761621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.217.3:35075 (local address 127.16.217.62:46265)
0620 20:20:57.761801 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0620 20:20:57.761805 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:20:57.763295 (+ 1490us) server_negotiation.cc:407] Connection header received
0620 20:20:57.765114 (+ 1819us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:20:57.765120 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:20:57.765207 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:20:57.765313 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:20:57.766393 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:57.767239 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:57.768987 (+ 1748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:57.769419 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:57.769565 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:20:57.770879 (+ 1314us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:20:57.770885 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:20:57.770890 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:20:57.770911 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:20:57.770940 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:20:57.770952 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:20:57.771041 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0620 20:20:57.771277 (+ 236us) server_negotiation.cc:299] Negotiation successful
0620 20:20:57.771367 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 20:20:57.772655 5116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:57.762221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:46265 (local address 127.16.217.3:35075)
0620 20:20:57.763149 (+ 928us) negotiation.cc:107] Waiting for socket to connect
0620 20:20:57.763189 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 20:20:57.764850 (+ 1661us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:20:57.765538 (+ 688us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:20:57.765545 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:20:57.765629 (+ 84us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:20:57.766228 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:20:57.766248 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:20:57.767385 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:20:57.767390 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:20:57.768847 (+ 1457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:20:57.768861 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:20:57.769632 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:20:57.769640 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:20:57.769909 (+ 269us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:20:57.770588 (+ 679us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 20:20:57.770590 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:20:57.770631 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 20:20:57.770699 (+ 68us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:20:57.771039 (+ 340us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:20:57.771067 (+ 28us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:20:57.771075 (+ 8us) client_negotiation.cc:769] Sending connection context
0620 20:20:57.771251 (+ 176us) client_negotiation.cc:240] Negotiation successful
0620 20:20:57.771614 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":491,"thread_start_us":301,"threads_started":1}
I20260620 20:20:57.773973 5114 heartbeater.cc:347] Connected to a master server at 127.16.217.62:46265
I20260620 20:20:57.774297 5114 heartbeater.cc:464] Registering TS with master...
I20260620 20:20:57.774972 5114 heartbeater.cc:511] Master 127.16.217.62:46265 requested a full tablet report, sending...
I20260620 20:20:57.776324 4667 ts_manager.cc:194] Registered new tserver with Master: d490a6234f3e41bea8127b1de701d889 (127.16.217.3:44509)
I20260620 20:20:57.777057 4667 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.217.3:35075
I20260620 20:20:57.782493 17252 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 20:20:57.792322 4855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:57.786571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44392 (local address 127.16.217.62:46265)
0620 20:20:57.786797 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0620 20:20:57.786804 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:20:57.787153 (+ 349us) server_negotiation.cc:407] Connection header received
0620 20:20:57.787335 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:20:57.787341 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:20:57.787417 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:20:57.787532 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:20:57.788071 (+ 539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:57.788773 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:57.790124 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:57.790541 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:57.790672 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:20:57.791653 (+ 981us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:20:57.791658 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:20:57.791661 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:20:57.791676 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:20:57.791705 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:20:57.791716 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:20:57.791805 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0620 20:20:57.791956 (+ 151us) server_negotiation.cc:299] Negotiation successful
0620 20:20:57.792041 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 20:20:57.796252 4667 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44392:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 20:20:57.803287 4667 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 20:20:57.830904 5128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:57.815691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.1:34185 (local address 127.0.0.1:54976)
0620 20:20:57.818392 (+ 2701us) negotiation.cc:107] Waiting for socket to connect
0620 20:20:57.818409 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 20:20:57.818543 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:20:57.819727 (+ 1184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:20:57.819739 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:20:57.819790 (+ 51us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:20:57.820271 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:20:57.820287 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:20:57.822823 (+ 2536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:20:57.822832 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:20:57.826605 (+ 3773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:20:57.826626 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:20:57.829570 (+ 2944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:20:57.829579 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:20:57.829898 (+ 319us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:20:57.829991 (+ 93us) client_negotiation.cc:769] Sending connection context
0620 20:20:57.830161 (+ 170us) client_negotiation.cc:240] Negotiation successful
0620 20:20:57.830421 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2562,"thread_start_us":2953,"threads_started":1}
I20260620 20:20:57.831166 5129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:57.816255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54976 (local address 127.16.217.1:34185)
0620 20:20:57.817957 (+ 1702us) server_negotiation.cc:206] Beginning negotiation
0620 20:20:57.817969 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 20:20:57.818698 (+ 729us) server_negotiation.cc:407] Connection header received
0620 20:20:57.818828 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:20:57.818839 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:20:57.819161 (+ 322us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:20:57.819382 (+ 221us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:20:57.820528 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:57.822620 (+ 2092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:57.826896 (+ 4276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:57.828899 (+ 2003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:57.829074 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:20:57.829402 (+ 328us) server_negotiation.cc:1035] Waiting for connection context
0620 20:20:57.830564 (+ 1162us) server_negotiation.cc:299] Negotiation successful
0620 20:20:57.830737 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1464,"thread_start_us":1331,"threads_started":1}
I20260620 20:20:57.835484 4788 tablet_service.cc:1467] Processing CreateTablet for tablet b2b70d1061a4407a877da2ded0951230 (DEFAULT_TABLE table=test-table [id=fa4a25217ebc4a80aa2ee38f777be113]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:20:57.836971 4788 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b2b70d1061a4407a877da2ded0951230. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:20:57.838063 5130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:57.817637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.3:44509 (local address 127.0.0.1:37690)
0620 20:20:57.830223 (+ 12586us) negotiation.cc:107] Waiting for socket to connect
0620 20:20:57.830237 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 20:20:57.830377 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:20:57.831282 (+ 905us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:20:57.831289 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:20:57.831319 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:20:57.831718 (+ 399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:20:57.831732 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:20:57.833670 (+ 1938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:20:57.833675 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:20:57.835613 (+ 1938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:20:57.835625 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:20:57.837271 (+ 1646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:20:57.837278 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:20:57.837480 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:20:57.837511 (+ 31us) client_negotiation.cc:769] Sending connection context
0620 20:20:57.837618 (+ 107us) client_negotiation.cc:240] Negotiation successful
0620 20:20:57.837747 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12463,"thread_start_us":12318,"threads_started":1}
I20260620 20:20:57.838251 5131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:57.818318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37690 (local address 127.16.217.3:44509)
0620 20:20:57.829561 (+ 11243us) server_negotiation.cc:206] Beginning negotiation
0620 20:20:57.829573 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 20:20:57.830347 (+ 774us) server_negotiation.cc:407] Connection header received
0620 20:20:57.830530 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:20:57.830538 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:20:57.830789 (+ 251us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:20:57.830991 (+ 202us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:20:57.831987 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:57.833517 (+ 1530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:57.835795 (+ 2278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:57.836796 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:57.836955 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:20:57.837261 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0620 20:20:57.837688 (+ 427us) server_negotiation.cc:299] Negotiation successful
0620 20:20:57.837846 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10998,"thread_start_us":11252,"threads_started":1}
I20260620 20:20:57.839856 5126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:57.815346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.2:46037 (local address 127.0.0.1:38606)
0620 20:20:57.816243 (+ 897us) negotiation.cc:107] Waiting for socket to connect
0620 20:20:57.816290 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 20:20:57.816524 (+ 234us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:20:57.832271 (+ 15747us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:20:57.832275 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:20:57.832303 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:20:57.832760 (+ 457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:20:57.832771 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:20:57.834631 (+ 1860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:20:57.834635 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:20:57.836712 (+ 2077us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:20:57.836728 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:20:57.839094 (+ 2366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:20:57.839101 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:20:57.839274 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:20:57.839308 (+ 34us) client_negotiation.cc:769] Sending connection context
0620 20:20:57.839398 (+ 90us) client_negotiation.cc:240] Negotiation successful
0620 20:20:57.839546 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":555,"mutex_wait_us":1,"thread_start_us":313,"threads_started":1}
I20260620 20:20:57.839941 5127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:57.815527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38606 (local address 127.16.217.2:46037)
0620 20:20:57.831697 (+ 16170us) server_negotiation.cc:206] Beginning negotiation
0620 20:20:57.831706 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 20:20:57.831734 (+ 28us) server_negotiation.cc:407] Connection header received
0620 20:20:57.831828 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:20:57.831834 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:20:57.832037 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:20:57.832974 (+ 937us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:20:57.833198 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:57.834475 (+ 1277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:57.836915 (+ 2440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:57.838857 (+ 1942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:57.839025 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:20:57.839269 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0620 20:20:57.839458 (+ 189us) server_negotiation.cc:299] Negotiation successful
0620 20:20:57.839624 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16000,"thread_start_us":15878,"threads_started":1}
I20260620 20:20:57.841843 5049 tablet_service.cc:1467] Processing CreateTablet for tablet b2b70d1061a4407a877da2ded0951230 (DEFAULT_TABLE table=test-table [id=fa4a25217ebc4a80aa2ee38f777be113]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:20:57.843319 5049 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b2b70d1061a4407a877da2ded0951230. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:20:57.843289 4919 tablet_service.cc:1467] Processing CreateTablet for tablet b2b70d1061a4407a877da2ded0951230 (DEFAULT_TABLE table=test-table [id=fa4a25217ebc4a80aa2ee38f777be113]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:20:57.844547 4919 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b2b70d1061a4407a877da2ded0951230. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:20:57.849623 5132 tablet_bootstrap.cc:492] T b2b70d1061a4407a877da2ded0951230 P 1c9e1103c5b14ed08970a95305274dc8: Bootstrap starting.
I20260620 20:20:57.853303 5132 tablet_bootstrap.cc:654] T b2b70d1061a4407a877da2ded0951230 P 1c9e1103c5b14ed08970a95305274dc8: Neither blocks nor log segments found. Creating new log.
I20260620 20:20:57.855119 5132 log.cc:824] T b2b70d1061a4407a877da2ded0951230 P 1c9e1103c5b14ed08970a95305274dc8: Log is configured to *not* fsync() on all Append() calls
I20260620 20:20:57.863843 5135 tablet_bootstrap.cc:492] T b2b70d1061a4407a877da2ded0951230 P a75146fcda6c47328e38eb51dbb35a07: Bootstrap starting.
I20260620 20:20:57.863451 5134 tablet_bootstrap.cc:492] T b2b70d1061a4407a877da2ded0951230 P d490a6234f3e41bea8127b1de701d889: Bootstrap starting.
I20260620 20:20:57.865073 5132 tablet_bootstrap.cc:492] T b2b70d1061a4407a877da2ded0951230 P 1c9e1103c5b14ed08970a95305274dc8: No bootstrap required, opened a new log
I20260620 20:20:57.865393 5132 ts_tablet_manager.cc:1397] T b2b70d1061a4407a877da2ded0951230 P 1c9e1103c5b14ed08970a95305274dc8: Time spent bootstrapping tablet: real 0.016s user 0.008s sys 0.006s
I20260620 20:20:57.868420 5134 tablet_bootstrap.cc:654] T b2b70d1061a4407a877da2ded0951230 P d490a6234f3e41bea8127b1de701d889: Neither blocks nor log segments found. Creating new log.
I20260620 20:20:57.869530 5135 tablet_bootstrap.cc:654] T b2b70d1061a4407a877da2ded0951230 P a75146fcda6c47328e38eb51dbb35a07: Neither blocks nor log segments found. Creating new log.
I20260620 20:20:57.869637 5134 log.cc:824] T b2b70d1061a4407a877da2ded0951230 P d490a6234f3e41bea8127b1de701d889: Log is configured to *not* fsync() on all Append() calls
I20260620 20:20:57.870702 5135 log.cc:824] T b2b70d1061a4407a877da2ded0951230 P a75146fcda6c47328e38eb51dbb35a07: Log is configured to *not* fsync() on all Append() calls
I20260620 20:20:57.872566 5134 tablet_bootstrap.cc:492] T b2b70d1061a4407a877da2ded0951230 P d490a6234f3e41bea8127b1de701d889: No bootstrap required, opened a new log
I20260620 20:20:57.872831 5134 ts_tablet_manager.cc:1397] T b2b70d1061a4407a877da2ded0951230 P d490a6234f3e41bea8127b1de701d889: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20260620 20:20:57.873548 5135 tablet_bootstrap.cc:492] T b2b70d1061a4407a877da2ded0951230 P a75146fcda6c47328e38eb51dbb35a07: No bootstrap required, opened a new log
I20260620 20:20:57.873790 5135 ts_tablet_manager.cc:1397] T b2b70d1061a4407a877da2ded0951230 P a75146fcda6c47328e38eb51dbb35a07: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.005s
I20260620 20:20:57.874006 5132 raft_consensus.cc:348] T b2b70d1061a4407a877da2ded0951230 P 1c9e1103c5b14ed08970a95305274dc8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a75146fcda6c47328e38eb51dbb35a07" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 46037 } } peers { permanent_uuid: "1c9e1103c5b14ed08970a95305274dc8" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 34185 } } peers { permanent_uuid: "d490a6234f3e41bea8127b1de701d889" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 44509 } }
I20260620 20:20:57.874353 5132 raft_consensus.cc:374] T b2b70d1061a4407a877da2ded0951230 P 1c9e1103c5b14ed08970a95305274dc8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:20:57.874476 5132 raft_consensus.cc:729] T b2b70d1061a4407a877da2ded0951230 P 1c9e1103c5b14ed08970a95305274dc8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c9e1103c5b14ed08970a95305274dc8, State: Initialized, Role: FOLLOWER
I20260620 20:20:57.875069 5132 consensus_queue.cc:260] T b2b70d1061a4407a877da2ded0951230 P 1c9e1103c5b14ed08970a95305274dc8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a75146fcda6c47328e38eb51dbb35a07" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 46037 } } peers { permanent_uuid: "1c9e1103c5b14ed08970a95305274dc8" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 34185 } } peers { permanent_uuid: "d490a6234f3e41bea8127b1de701d889" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 44509 } }
I20260620 20:20:57.876116 4853 heartbeater.cc:503] Master 127.16.217.62:46265 was elected leader, sending a full tablet report...
I20260620 20:20:57.876780 5132 ts_tablet_manager.cc:1428] T b2b70d1061a4407a877da2ded0951230 P 1c9e1103c5b14ed08970a95305274dc8: Time spent starting tablet: real 0.011s user 0.010s sys 0.000s
I20260620 20:20:57.879810 5138 raft_consensus.cc:482] T b2b70d1061a4407a877da2ded0951230 P 1c9e1103c5b14ed08970a95305274dc8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 20:20:57.880541 5138 raft_consensus.cc:504] T b2b70d1061a4407a877da2ded0951230 P 1c9e1103c5b14ed08970a95305274dc8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a75146fcda6c47328e38eb51dbb35a07" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 46037 } } peers { permanent_uuid: "1c9e1103c5b14ed08970a95305274dc8" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 34185 } } peers { permanent_uuid: "d490a6234f3e41bea8127b1de701d889" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 44509 } }
I20260620 20:20:57.883554 5134 raft_consensus.cc:348] T b2b70d1061a4407a877da2ded0951230 P d490a6234f3e41bea8127b1de701d889 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a75146fcda6c47328e38eb51dbb35a07" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 46037 } } peers { permanent_uuid: "1c9e1103c5b14ed08970a95305274dc8" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 34185 } } peers { permanent_uuid: "d490a6234f3e41bea8127b1de701d889" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 44509 } }
I20260620 20:20:57.884080 5134 raft_consensus.cc:374] T b2b70d1061a4407a877da2ded0951230 P d490a6234f3e41bea8127b1de701d889 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:20:57.884186 5134 raft_consensus.cc:729] T b2b70d1061a4407a877da2ded0951230 P d490a6234f3e41bea8127b1de701d889 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d490a6234f3e41bea8127b1de701d889, State: Initialized, Role: FOLLOWER
I20260620 20:20:57.884914 5138 leader_election.cc:288] T b2b70d1061a4407a877da2ded0951230 P 1c9e1103c5b14ed08970a95305274dc8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a75146fcda6c47328e38eb51dbb35a07 (127.16.217.2:46037), d490a6234f3e41bea8127b1de701d889 (127.16.217.3:44509)
I20260620 20:20:57.884789 5134 consensus_queue.cc:260] T b2b70d1061a4407a877da2ded0951230 P d490a6234f3e41bea8127b1de701d889 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a75146fcda6c47328e38eb51dbb35a07" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 46037 } } peers { permanent_uuid: "1c9e1103c5b14ed08970a95305274dc8" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 34185 } } peers { permanent_uuid: "d490a6234f3e41bea8127b1de701d889" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 44509 } }
I20260620 20:20:57.884949 5135 raft_consensus.cc:348] T b2b70d1061a4407a877da2ded0951230 P a75146fcda6c47328e38eb51dbb35a07 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a75146fcda6c47328e38eb51dbb35a07" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 46037 } } peers { permanent_uuid: "1c9e1103c5b14ed08970a95305274dc8" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 34185 } } peers { permanent_uuid: "d490a6234f3e41bea8127b1de701d889" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 44509 } }
I20260620 20:20:57.885313 5135 raft_consensus.cc:374] T b2b70d1061a4407a877da2ded0951230 P a75146fcda6c47328e38eb51dbb35a07 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:20:57.885406 5135 raft_consensus.cc:729] T b2b70d1061a4407a877da2ded0951230 P a75146fcda6c47328e38eb51dbb35a07 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a75146fcda6c47328e38eb51dbb35a07, State: Initialized, Role: FOLLOWER
I20260620 20:20:57.886056 5135 consensus_queue.cc:260] T b2b70d1061a4407a877da2ded0951230 P a75146fcda6c47328e38eb51dbb35a07 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a75146fcda6c47328e38eb51dbb35a07" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 46037 } } peers { permanent_uuid: "1c9e1103c5b14ed08970a95305274dc8" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 34185 } } peers { permanent_uuid: "d490a6234f3e41bea8127b1de701d889" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 44509 } }
I20260620 20:20:57.893101 5114 heartbeater.cc:503] Master 127.16.217.62:46265 was elected leader, sending a full tablet report...
I20260620 20:20:57.893448 5134 ts_tablet_manager.cc:1428] T b2b70d1061a4407a877da2ded0951230 P d490a6234f3e41bea8127b1de701d889: Time spent starting tablet: real 0.020s user 0.016s sys 0.004s
I20260620 20:20:57.894389 5135 ts_tablet_manager.cc:1428] T b2b70d1061a4407a877da2ded0951230 P a75146fcda6c47328e38eb51dbb35a07: Time spent starting tablet: real 0.020s user 0.015s sys 0.004s
I20260620 20:20:57.894805 4984 heartbeater.cc:503] Master 127.16.217.62:46265 was elected leader, sending a full tablet report...
I20260620 20:20:57.895654 5127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:57.883875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.217.1:41607 (local address 127.16.217.2:46037)
0620 20:20:57.884095 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0620 20:20:57.884099 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:20:57.884117 (+ 18us) server_negotiation.cc:407] Connection header received
0620 20:20:57.884197 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:20:57.884204 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:20:57.884303 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:20:57.884447 (+ 144us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:20:57.885854 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:57.887381 (+ 1527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:57.890852 (+ 3471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:57.892523 (+ 1671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:57.892690 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:20:57.892795 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0620 20:20:57.895097 (+ 2302us) server_negotiation.cc:299] Negotiation successful
0620 20:20:57.895295 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 20:20:57.896533 5139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:57.882468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.2:46037 (local address 127.16.217.1:41607)
0620 20:20:57.882949 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0620 20:20:57.882975 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0620 20:20:57.883092 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:20:57.885133 (+ 2041us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:20:57.885137 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:20:57.885162 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:20:57.885515 (+ 353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:20:57.885532 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:20:57.887601 (+ 2069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:20:57.887609 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:20:57.890583 (+ 2974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:20:57.890603 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:20:57.894554 (+ 3951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:20:57.894563 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:20:57.894835 (+ 272us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:20:57.894884 (+ 49us) client_negotiation.cc:769] Sending connection context
0620 20:20:57.895876 (+ 992us) client_negotiation.cc:240] Negotiation successful
0620 20:20:57.896098 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":2106,"threads_started":1}
I20260620 20:20:57.897593 4939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2b70d1061a4407a877da2ded0951230" candidate_uuid: "1c9e1103c5b14ed08970a95305274dc8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a75146fcda6c47328e38eb51dbb35a07" is_pre_election: true
I20260620 20:20:57.898207 4939 raft_consensus.cc:2454] T b2b70d1061a4407a877da2ded0951230 P a75146fcda6c47328e38eb51dbb35a07 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c9e1103c5b14ed08970a95305274dc8 in term 0.
I20260620 20:20:57.898936 4741 leader_election.cc:302] T b2b70d1061a4407a877da2ded0951230 P 1c9e1103c5b14ed08970a95305274dc8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c9e1103c5b14ed08970a95305274dc8, a75146fcda6c47328e38eb51dbb35a07; no voters:
I20260620 20:20:57.899350 5138 raft_consensus.cc:2781] T b2b70d1061a4407a877da2ded0951230 P 1c9e1103c5b14ed08970a95305274dc8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 20:20:57.899478 5138 raft_consensus.cc:482] T b2b70d1061a4407a877da2ded0951230 P 1c9e1103c5b14ed08970a95305274dc8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 20:20:57.899557 5138 raft_consensus.cc:3037] T b2b70d1061a4407a877da2ded0951230 P 1c9e1103c5b14ed08970a95305274dc8 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:20:57.902827 5138 raft_consensus.cc:504] T b2b70d1061a4407a877da2ded0951230 P 1c9e1103c5b14ed08970a95305274dc8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a75146fcda6c47328e38eb51dbb35a07" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 46037 } } peers { permanent_uuid: "1c9e1103c5b14ed08970a95305274dc8" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 34185 } } peers { permanent_uuid: "d490a6234f3e41bea8127b1de701d889" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 44509 } }
I20260620 20:20:57.903450 5138 leader_election.cc:288] T b2b70d1061a4407a877da2ded0951230 P 1c9e1103c5b14ed08970a95305274dc8 [CANDIDATE]: Term 1 election: Requested vote from peers a75146fcda6c47328e38eb51dbb35a07 (127.16.217.2:46037), d490a6234f3e41bea8127b1de701d889 (127.16.217.3:44509)
I20260620 20:20:57.903915 4939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2b70d1061a4407a877da2ded0951230" candidate_uuid: "1c9e1103c5b14ed08970a95305274dc8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a75146fcda6c47328e38eb51dbb35a07"
I20260620 20:20:57.904127 4939 raft_consensus.cc:3037] T b2b70d1061a4407a877da2ded0951230 P a75146fcda6c47328e38eb51dbb35a07 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:20:57.904505 5140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:57.885110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.3:44509 (local address 127.16.217.1:52761)
0620 20:20:57.897495 (+ 12385us) negotiation.cc:107] Waiting for socket to connect
0620 20:20:57.897510 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 20:20:57.897660 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:20:57.898220 (+ 560us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:20:57.898224 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:20:57.898251 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:20:57.898599 (+ 348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:20:57.898616 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:20:57.900119 (+ 1503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:20:57.900123 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:20:57.902272 (+ 2149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:20:57.902285 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:20:57.903701 (+ 1416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:20:57.903705 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:20:57.903897 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:20:57.903938 (+ 41us) client_negotiation.cc:769] Sending connection context
0620 20:20:57.904056 (+ 118us) client_negotiation.cc:240] Negotiation successful
0620 20:20:57.904193 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12279,"thread_start_us":12120,"threads_started":1}
I20260620 20:20:57.904531 5131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:57.885423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.217.1:52761 (local address 127.16.217.3:44509)
0620 20:20:57.885592 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0620 20:20:57.885597 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:20:57.897619 (+ 12022us) server_negotiation.cc:407] Connection header received
0620 20:20:57.897771 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:20:57.897776 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:20:57.897846 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:20:57.897955 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:20:57.898905 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:57.899934 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:57.902490 (+ 2556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:57.903449 (+ 959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:57.903698 (+ 249us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:20:57.903790 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0620 20:20:57.904105 (+ 315us) server_negotiation.cc:299] Negotiation successful
0620 20:20:57.904257 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 20:20:57.905668 5069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2b70d1061a4407a877da2ded0951230" candidate_uuid: "1c9e1103c5b14ed08970a95305274dc8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d490a6234f3e41bea8127b1de701d889" is_pre_election: true
I20260620 20:20:57.905726 5068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2b70d1061a4407a877da2ded0951230" candidate_uuid: "1c9e1103c5b14ed08970a95305274dc8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d490a6234f3e41bea8127b1de701d889"
I20260620 20:20:57.906206 5068 raft_consensus.cc:3037] T b2b70d1061a4407a877da2ded0951230 P d490a6234f3e41bea8127b1de701d889 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:20:57.907403 4939 raft_consensus.cc:2454] T b2b70d1061a4407a877da2ded0951230 P a75146fcda6c47328e38eb51dbb35a07 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c9e1103c5b14ed08970a95305274dc8 in term 1.
I20260620 20:20:57.907732 4741 leader_election.cc:302] T b2b70d1061a4407a877da2ded0951230 P 1c9e1103c5b14ed08970a95305274dc8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c9e1103c5b14ed08970a95305274dc8, a75146fcda6c47328e38eb51dbb35a07; no voters:
I20260620 20:20:57.908016 5138 raft_consensus.cc:2781] T b2b70d1061a4407a877da2ded0951230 P 1c9e1103c5b14ed08970a95305274dc8 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:20:57.908465 5138 raft_consensus.cc:686] T b2b70d1061a4407a877da2ded0951230 P 1c9e1103c5b14ed08970a95305274dc8 [term 1 LEADER]: Becoming Leader. State: Replica: 1c9e1103c5b14ed08970a95305274dc8, State: Running, Role: LEADER
I20260620 20:20:57.908972 5138 consensus_queue.cc:237] T b2b70d1061a4407a877da2ded0951230 P 1c9e1103c5b14ed08970a95305274dc8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a75146fcda6c47328e38eb51dbb35a07" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 46037 } } peers { permanent_uuid: "1c9e1103c5b14ed08970a95305274dc8" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 34185 } } peers { permanent_uuid: "d490a6234f3e41bea8127b1de701d889" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 44509 } }
I20260620 20:20:57.909539 5068 raft_consensus.cc:2454] T b2b70d1061a4407a877da2ded0951230 P d490a6234f3e41bea8127b1de701d889 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c9e1103c5b14ed08970a95305274dc8 in term 1.
I20260620 20:20:57.913321 4667 catalog_manager.cc:5495] T b2b70d1061a4407a877da2ded0951230 P 1c9e1103c5b14ed08970a95305274dc8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1c9e1103c5b14ed08970a95305274dc8 (127.16.217.1). New cstate: current_term: 1 leader_uuid: "1c9e1103c5b14ed08970a95305274dc8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a75146fcda6c47328e38eb51dbb35a07" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 46037 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c9e1103c5b14ed08970a95305274dc8" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 34185 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d490a6234f3e41bea8127b1de701d889" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 44509 } health_report { overall_health: UNKNOWN } } }
I20260620 20:20:57.950680 5129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:57.945305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54990 (local address 127.16.217.1:34185)
0620 20:20:57.945545 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0620 20:20:57.945551 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:20:57.945569 (+ 18us) server_negotiation.cc:407] Connection header received
0620 20:20:57.945677 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:20:57.945680 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:20:57.945768 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:20:57.945947 (+ 179us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 20:20:57.946345 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:57.947041 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:57.948529 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:57.948959 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:57.949101 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:20:57.949564 (+ 463us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 20:20:57.950128 (+ 564us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 20:20:57.950284 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0620 20:20:57.950343 (+ 59us) server_negotiation.cc:299] Negotiation successful
0620 20:20:57.950417 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260620 20:20:57.958817 5068 raft_consensus.cc:1261] T b2b70d1061a4407a877da2ded0951230 P d490a6234f3e41bea8127b1de701d889 [term 1 FOLLOWER]: Refusing update from remote peer 1c9e1103c5b14ed08970a95305274dc8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:20:57.958917 4939 raft_consensus.cc:1261] T b2b70d1061a4407a877da2ded0951230 P a75146fcda6c47328e38eb51dbb35a07 [term 1 FOLLOWER]: Refusing update from remote peer 1c9e1103c5b14ed08970a95305274dc8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:20:57.959791 5143 consensus_queue.cc:1035] T b2b70d1061a4407a877da2ded0951230 P 1c9e1103c5b14ed08970a95305274dc8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a75146fcda6c47328e38eb51dbb35a07" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 46037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:20:57.960188 5138 consensus_queue.cc:1035] T b2b70d1061a4407a877da2ded0951230 P 1c9e1103c5b14ed08970a95305274dc8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d490a6234f3e41bea8127b1de701d889" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 44509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:20:57.982662 4667 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44392:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 20:20:57.982884 4667 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44392:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 20:20:57.986478 4667 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e76f35a6e1134497b75e69e38296cc7c: Sending DeleteTablet for 3 replicas of tablet b2b70d1061a4407a877da2ded0951230
I20260620 20:20:57.987648 4919 tablet_service.cc:1514] Processing DeleteTablet for tablet b2b70d1061a4407a877da2ded0951230 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:20:57 UTC) from {username='slave'} at 127.0.0.1:38606
I20260620 20:20:57.987746 4788 tablet_service.cc:1514] Processing DeleteTablet for tablet b2b70d1061a4407a877da2ded0951230 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:20:57 UTC) from {username='slave'} at 127.0.0.1:54976
I20260620 20:20:57.987895 5049 tablet_service.cc:1514] Processing DeleteTablet for tablet b2b70d1061a4407a877da2ded0951230 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:20:57 UTC) from {username='slave'} at 127.0.0.1:37690
I20260620 20:20:57.989777 17252 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu pid 4726
I20260620 20:20:58.017894 5156 tablet_replica.cc:330] T b2b70d1061a4407a877da2ded0951230 P 1c9e1103c5b14ed08970a95305274dc8: stopping tablet replica
I20260620 20:20:58.023449 5156 raft_consensus.cc:2229] T b2b70d1061a4407a877da2ded0951230 P 1c9e1103c5b14ed08970a95305274dc8 [term 1 LEADER]: Raft consensus shutting down.
I20260620 20:20:58.024477 5156 raft_consensus.cc:2258] T b2b70d1061a4407a877da2ded0951230 P 1c9e1103c5b14ed08970a95305274dc8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:20:58.029536 5156 ts_tablet_manager.cc:1905] T b2b70d1061a4407a877da2ded0951230 P 1c9e1103c5b14ed08970a95305274dc8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:20:58.032846 5129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:57.992358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55000 (local address 127.16.217.1:34185)
0620 20:20:57.992911 (+ 553us) server_negotiation.cc:206] Beginning negotiation
0620 20:20:57.992920 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 20:20:58.015392 (+ 22472us) server_negotiation.cc:407] Connection header received
0620 20:20:58.016018 (+ 626us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:20:58.016028 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:20:58.016214 (+ 186us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:20:58.016325 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:20:58.018213 (+ 1888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:58.019813 (+ 1600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:58.025149 (+ 5336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:58.026061 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:58.026169 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:20:58.030247 (+ 4078us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:20:58.030262 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:20:58.030286 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:20:58.031282 (+ 996us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:20:58.031357 (+ 75us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:20:58.031412 (+ 55us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:20:58.031537 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0620 20:20:58.032185 (+ 648us) server_negotiation.cc:299] Negotiation successful
0620 20:20:58.032330 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":384}
I20260620 20:20:58.034547 4848 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:20:58.035385 5155 tablet_replica.cc:330] T b2b70d1061a4407a877da2ded0951230 P a75146fcda6c47328e38eb51dbb35a07: stopping tablet replica
I20260620 20:20:58.035924 5155 raft_consensus.cc:2229] T b2b70d1061a4407a877da2ded0951230 P a75146fcda6c47328e38eb51dbb35a07 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 20:20:58.036549 5155 raft_consensus.cc:2258] T b2b70d1061a4407a877da2ded0951230 P a75146fcda6c47328e38eb51dbb35a07 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:20:58.037462 5157 tablet_replica.cc:330] T b2b70d1061a4407a877da2ded0951230 P d490a6234f3e41bea8127b1de701d889: stopping tablet replica
I20260620 20:20:58.037959 5157 raft_consensus.cc:2229] T b2b70d1061a4407a877da2ded0951230 P d490a6234f3e41bea8127b1de701d889 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 20:20:58.038540 5157 raft_consensus.cc:2258] T b2b70d1061a4407a877da2ded0951230 P d490a6234f3e41bea8127b1de701d889 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:20:58.038950 5155 ts_tablet_manager.cc:1905] T b2b70d1061a4407a877da2ded0951230 P a75146fcda6c47328e38eb51dbb35a07: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:20:58.040601 5157 ts_tablet_manager.cc:1905] T b2b70d1061a4407a877da2ded0951230 P d490a6234f3e41bea8127b1de701d889: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:20:58.045715 5157 ts_tablet_manager.cc:1918] T b2b70d1061a4407a877da2ded0951230 P d490a6234f3e41bea8127b1de701d889: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 20:20:58.045881 5157 log.cc:1196] T b2b70d1061a4407a877da2ded0951230 P d490a6234f3e41bea8127b1de701d889: Deleting WAL directory at /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986536687152-17252-0/minicluster-data/ts-2/wal/wals/b2b70d1061a4407a877da2ded0951230
I20260620 20:20:58.046356 5157 ts_tablet_manager.cc:1939] T b2b70d1061a4407a877da2ded0951230 P d490a6234f3e41bea8127b1de701d889: Deleting consensus metadata
I20260620 20:20:58.046936 5155 ts_tablet_manager.cc:1918] T b2b70d1061a4407a877da2ded0951230 P a75146fcda6c47328e38eb51dbb35a07: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 20:20:58.047098 5155 log.cc:1196] T b2b70d1061a4407a877da2ded0951230 P a75146fcda6c47328e38eb51dbb35a07: Deleting WAL directory at /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986536687152-17252-0/minicluster-data/ts-1/wal/wals/b2b70d1061a4407a877da2ded0951230
I20260620 20:20:58.047676 5155 ts_tablet_manager.cc:1939] T b2b70d1061a4407a877da2ded0951230 P a75146fcda6c47328e38eb51dbb35a07: Deleting consensus metadata
I20260620 20:20:58.048197 4653 catalog_manager.cc:4841] TS d490a6234f3e41bea8127b1de701d889 (127.16.217.3:44509): tablet b2b70d1061a4407a877da2ded0951230 (table test-table [id=fa4a25217ebc4a80aa2ee38f777be113]) successfully deleted
I20260620 20:20:58.049345 4651 catalog_manager.cc:4841] TS a75146fcda6c47328e38eb51dbb35a07 (127.16.217.2:46037): tablet b2b70d1061a4407a877da2ded0951230 (table test-table [id=fa4a25217ebc4a80aa2ee38f777be113]) successfully deleted
I20260620 20:20:58.137396 17252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu with pid 4726
W20260620 20:20:58.156574 4652 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.16.217.1:34185 (error 108)
I20260620 20:20:58.157318 17252 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu pid 4857
I20260620 20:20:58.158478 5126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:58.158038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.1:34185 (local address 127.0.0.1:55010)
0620 20:20:58.158214 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0620 20:20:58.158347 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.217.1:34185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260620 20:20:58.158787 4652 catalog_manager.cc:4584] TS 1c9e1103c5b14ed08970a95305274dc8 (127.16.217.1:34185): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b2b70d1061a4407a877da2ded0951230: Network error: Client connection negotiation failed: client connection to 127.16.217.1:34185: connect: Connection refused (error 111)
I20260620 20:20:58.164292 5127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:58.158212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38614 (local address 127.16.217.2:46037)
0620 20:20:58.158402 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0620 20:20:58.158408 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:20:58.158428 (+ 20us) server_negotiation.cc:407] Connection header received
0620 20:20:58.158700 (+ 272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:20:58.158708 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:20:58.158840 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:20:58.158941 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:20:58.159433 (+ 492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:58.160268 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:58.161411 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:58.161803 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:58.161954 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:20:58.162985 (+ 1031us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:20:58.162993 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:20:58.163009 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:20:58.163627 (+ 618us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:20:58.163670 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:20:58.163689 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:20:58.163779 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0620 20:20:58.163916 (+ 137us) server_negotiation.cc:299] Negotiation successful
0620 20:20:58.164006 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 20:20:58.164937 4979 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:20:58.212538 5126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:58.212058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.1:34185 (local address 127.0.0.1:55016)
0620 20:20:58.212275 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0620 20:20:58.212368 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.217.1:34185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260620 20:20:58.213593 5126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:58.213242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.1:34185 (local address 127.0.0.1:55030)
0620 20:20:58.213392 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0620 20:20:58.213460 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.217.1:34185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260620 20:20:58.252732 17252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu with pid 4857
I20260620 20:20:58.271658 17252 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu pid 4987
I20260620 20:20:58.274178 5126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:58.273707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.1:34185 (local address 127.0.0.1:55040)
0620 20:20:58.273945 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0620 20:20:58.274026 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.217.1:34185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260620 20:20:58.275146 5126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:58.274784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.1:34185 (local address 127.0.0.1:55042)
0620 20:20:58.274931 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0620 20:20:58.275032 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.217.1:34185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260620 20:20:58.278856 5131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:58.272424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37698 (local address 127.16.217.3:44509)
0620 20:20:58.272616 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0620 20:20:58.272622 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:20:58.272692 (+ 70us) server_negotiation.cc:407] Connection header received
0620 20:20:58.272904 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:20:58.272913 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:20:58.273028 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:20:58.273134 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:20:58.273640 (+ 506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:58.274495 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:58.275827 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:58.276238 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:58.276364 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:20:58.277406 (+ 1042us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:20:58.277415 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:20:58.277431 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:20:58.278159 (+ 728us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:20:58.278207 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:20:58.278232 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:20:58.278352 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0620 20:20:58.278476 (+ 124us) server_negotiation.cc:299] Negotiation successful
0620 20:20:58.278582 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 20:20:58.279498 5109 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:20:58.355648 5126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:58.355177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.1:34185 (local address 127.0.0.1:55050)
0620 20:20:58.355389 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0620 20:20:58.355500 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.217.1:34185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260620 20:20:58.356772 5126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:58.356372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.1:34185 (local address 127.0.0.1:55056)
0620 20:20:58.356597 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0620 20:20:58.356666 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.217.1:34185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
I20260620 20:20:58.368692 17252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu with pid 4987
I20260620 20:20:58.385682 17252 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu pid 4636
I20260620 20:20:58.386809 4697 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:20:58.496795 17252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu with pid 4636
2026-06-20T20:20:58Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (2727 ms)
[----------] 6 tests from EncryptionPolicyTest (22022 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781986536687152-17252-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 20:20:58 dist-test-slave-2hpg krb5kdc[5166](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 20:20:58 dist-test-slave-2hpg krb5kdc[5166](info): set up 2 sockets
Jun 20 20:20:58 dist-test-slave-2hpg krb5kdc[5166](info): commencing operation
krb5kdc: starting...
W20260620 20:20:59.614761 17252 mini_kdc.cc:121] Time spent starting KDC: real 1.074s user 0.003s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781986536687152-17252-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781986536687152-17252-0/krb5kdc/test-user.keytab.
Jun 20 20:20:59 dist-test-slave-2hpg krb5kdc[5166](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986859, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T20:20:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T20:20:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781986536687152-17252-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781986536687152-17252-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 20:20:59.862179 17252 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
/tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781986536687152-17252-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781986536687152-17252-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781986536687152-17252-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781986536687152-17252-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37189
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781986536687152-17252-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42965
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37189
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781986536687152-17252-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781986536687152-17252-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781986536687152-17252-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:21:00.190138 5179 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:21:00.190415 5179 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:21:00.190578 5179 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:21:00.197712 5179 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:21:00.197810 5179 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:21:00.197927 5179 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:21:00.198056 5179 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:21:00.198119 5179 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:21:00.207861 5179 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42965
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781986536687152-17252-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781986536687152-17252-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37189
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781986536687152-17252-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37189
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781986536687152-17252-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781986536687152-17252-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:21:00.209157 5179 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:21:00.210507 5179 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:21:00.222489 5184 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:21:00.222581 5187 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:21:00.222435 5185 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:21:00.223660 5179 server_base.cc:1034] running on GCE node
I20260620 20:21:00.224485 5179 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:21:00.226035 5179 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:21:00.227312 5179 hybrid_clock.cc:648] HybridClock initialized: now 1781986860227262 us; error 56 us; skew 500 ppm
I20260620 20:21:00.227699 5179 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:21:00 dist-test-slave-2hpg krb5kdc[5166](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986860, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:21:00.230839 5179 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 20:21:00.232358 5179 webserver.cc:460] Webserver started at http://127.0.0.1:35119/ using document root <none> and password file <none>
I20260620 20:21:00.233032 5179 fs_manager.cc:359] Metadata directory not provided
I20260620 20:21:00.233179 5179 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:21:00.233479 5179 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:21:00.236516 5179 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781986536687152-17252-0/minicluster-data/master-0/data/instance:
uuid: "3b48614e0af543b884ed50c287644938"
format_stamp: "Formatted at 2026-06-20 20:21:00 on dist-test-slave-2hpg"
server_key: "1181438412e80ad22ac103f54dac55f9"
server_key_iv: "db8a05a57ee1fa43ed2d1ae978ba9f17"
server_key_version: "encryptionkey@0"
I20260620 20:21:00.237286 5179 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781986536687152-17252-0/minicluster-data/master-0/wal/instance:
uuid: "3b48614e0af543b884ed50c287644938"
format_stamp: "Formatted at 2026-06-20 20:21:00 on dist-test-slave-2hpg"
server_key: "1181438412e80ad22ac103f54dac55f9"
server_key_iv: "db8a05a57ee1fa43ed2d1ae978ba9f17"
server_key_version: "encryptionkey@0"
I20260620 20:21:00.242377 5179 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260620 20:21:00.245812 5194 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:21:00.247164 5179 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 20:21:00.247354 5179 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781986536687152-17252-0/minicluster-data/master-0/data,/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781986536687152-17252-0/minicluster-data/master-0/wal
uuid: "3b48614e0af543b884ed50c287644938"
format_stamp: "Formatted at 2026-06-20 20:21:00 on dist-test-slave-2hpg"
server_key: "1181438412e80ad22ac103f54dac55f9"
server_key_iv: "db8a05a57ee1fa43ed2d1ae978ba9f17"
server_key_version: "encryptionkey@0"
I20260620 20:21:00.247596 5179 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781986536687152-17252-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781986536687152-17252-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781986536687152-17252-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:21:00.278072 5179 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:21:00.282599 5179 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:21:00.282907 5179 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:21:00.302371 5179 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37189
I20260620 20:21:00.302459 5245 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37189 every 8 connection(s)
I20260620 20:21:00.303774 5179 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781986536687152-17252-0/minicluster-data/master-0/data/info.pb
I20260620 20:21:00.305374 17252 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu as pid 5179
I20260620 20:21:00.305678 17252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781986536687152-17252-0/minicluster-data/master-0/wal/instance
I20260620 20:21:00.306609 17252 external_mini_cluster.cc:1428] Setting key 3bab69ae38c220f800eb29df67867fd3
I20260620 20:21:00.308900 5246 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 20 20:21:00 dist-test-slave-2hpg krb5kdc[5166](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986859, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 20:21:00.323618 5246 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3b48614e0af543b884ed50c287644938: Bootstrap starting.
I20260620 20:21:00.327556 5246 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3b48614e0af543b884ed50c287644938: Neither blocks nor log segments found. Creating new log.
I20260620 20:21:00.328965 5246 log.cc:824] T 00000000000000000000000000000000 P 3b48614e0af543b884ed50c287644938: Log is configured to *not* fsync() on all Append() calls
I20260620 20:21:00.332170 5249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:00.308698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34373 (local address 127.0.0.1:37189)
0620 20:21:00.309572 (+ 874us) server_negotiation.cc:206] Beginning negotiation
0620 20:21:00.309585 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 20:21:00.309626 (+ 41us) server_negotiation.cc:407] Connection header received
0620 20:21:00.311345 (+ 1719us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:21:00.311367 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:21:00.311768 (+ 401us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:21:00.312098 (+ 330us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:21:00.314037 (+ 1939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:00.315421 (+ 1384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:00.317361 (+ 1940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:00.318185 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:00.318324 (+ 139us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:21:00.323903 (+ 5579us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:21:00.323912 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:21:00.323927 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:21:00.323966 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:21:00.327303 (+ 3337us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:21:00.328366 (+ 1063us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:21:00.328373 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:21:00.328379 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:21:00.328587 (+ 208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:21:00.329220 (+ 633us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:21:00.329224 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:21:00.329227 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:21:00.329971 (+ 744us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:21:00.330347 (+ 376us) server_negotiation.cc:1035] Waiting for connection context
0620 20:21:00.330830 (+ 483us) server_negotiation.cc:299] Negotiation successful
0620 20:21:00.331150 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":660,"thread_start_us":442,"threads_started":1}
I20260620 20:21:00.332372 5246 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3b48614e0af543b884ed50c287644938: No bootstrap required, opened a new log
I20260620 20:21:00.336438 17252 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 20:21:00.343235 5246 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3b48614e0af543b884ed50c287644938 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b48614e0af543b884ed50c287644938" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37189 } }
I20260620 20:21:00.343804 5246 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3b48614e0af543b884ed50c287644938 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:21:00.343912 5246 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3b48614e0af543b884ed50c287644938 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b48614e0af543b884ed50c287644938, State: Initialized, Role: FOLLOWER
I20260620 20:21:00.344527 5246 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3b48614e0af543b884ed50c287644938 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b48614e0af543b884ed50c287644938" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37189 } }
I20260620 20:21:00.344774 5246 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3b48614e0af543b884ed50c287644938 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 20:21:00.344933 5246 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3b48614e0af543b884ed50c287644938 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 20:21:00.345125 5246 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3b48614e0af543b884ed50c287644938 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:21:00.348156 5246 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3b48614e0af543b884ed50c287644938 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b48614e0af543b884ed50c287644938" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37189 } }
I20260620 20:21:00.348803 5246 leader_election.cc:302] T 00000000000000000000000000000000 P 3b48614e0af543b884ed50c287644938 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3b48614e0af543b884ed50c287644938; no voters:
I20260620 20:21:00.349532 5246 leader_election.cc:288] T 00000000000000000000000000000000 P 3b48614e0af543b884ed50c287644938 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 20:21:00.349665 5256 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3b48614e0af543b884ed50c287644938 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:21:00.351922 5246 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3b48614e0af543b884ed50c287644938 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:21:00.352834 5256 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3b48614e0af543b884ed50c287644938 [term 1 LEADER]: Becoming Leader. State: Replica: 3b48614e0af543b884ed50c287644938, State: Running, Role: LEADER
I20260620 20:21:00.353832 5256 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3b48614e0af543b884ed50c287644938 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b48614e0af543b884ed50c287644938" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37189 } }
I20260620 20:21:00.360447 5257 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b48614e0af543b884ed50c287644938 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3b48614e0af543b884ed50c287644938" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b48614e0af543b884ed50c287644938" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37189 } } }
I20260620 20:21:00.360772 5257 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b48614e0af543b884ed50c287644938 [sys.catalog]: This master's current role is: LEADER
I20260620 20:21:00.361832 5258 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b48614e0af543b884ed50c287644938 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3b48614e0af543b884ed50c287644938. Latest consensus state: current_term: 1 leader_uuid: "3b48614e0af543b884ed50c287644938" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b48614e0af543b884ed50c287644938" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37189 } } }
I20260620 20:21:00.362144 5258 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b48614e0af543b884ed50c287644938 [sys.catalog]: This master's current role is: LEADER
I20260620 20:21:00.362782 5249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:00.342624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36945 (local address 127.0.0.1:37189)
0620 20:21:00.342816 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0620 20:21:00.342823 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:21:00.343298 (+ 475us) server_negotiation.cc:407] Connection header received
0620 20:21:00.343422 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:21:00.343427 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:21:00.343517 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:21:00.343639 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:21:00.345388 (+ 1749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:00.346667 (+ 1279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:00.347849 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:00.348574 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:00.348756 (+ 182us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:21:00.352670 (+ 3914us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:21:00.352677 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:21:00.352686 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:21:00.352727 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:21:00.357429 (+ 4702us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:21:00.358597 (+ 1168us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:21:00.358601 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:21:00.358603 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:21:00.358736 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:21:00.360164 (+ 1428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:21:00.360170 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:21:00.360173 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:21:00.360633 (+ 460us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:21:00.361784 (+ 1151us) server_negotiation.cc:1035] Waiting for connection context
0620 20:21:00.362069 (+ 285us) server_negotiation.cc:299] Negotiation successful
0620 20:21:00.362261 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 20:21:00.366755 5270 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 20:21:00.372404 5270 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 20:21:00.381151 5270 catalog_manager.cc:1347] Generated new cluster ID: 73d2ad0b610f443e875de514bc4518c3
I20260620 20:21:00.381304 5270 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 20:21:00.391331 5270 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 20:21:00.393131 5270 catalog_manager.cc:1504] Loading token signing keys...
I20260620 20:21:00.400947 5270 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3b48614e0af543b884ed50c287644938: Generated new TSK 0
I20260620 20:21:00.401952 5270 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 20:21:00.415118 17252 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu pid 5179
I20260620 20:21:00.417254 5241 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:21:00.507354 17252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu with pid 5179
W20260620 20:21:00.524930 5164 connection.cc:535] client connection to 127.0.0.1:37189 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:37189): Connection reset by peer (error 104)
2026-06-20T20:21:00Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (2004 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781986536687152-17252-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 20:21:00 dist-test-slave-2hpg krb5kdc[5278](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 20:21:00 dist-test-slave-2hpg krb5kdc[5278](info): set up 2 sockets
Jun 20 20:21:00 dist-test-slave-2hpg krb5kdc[5278](info): commencing operation
krb5kdc: starting...
W20260620 20:21:01.650411 17252 mini_kdc.cc:121] Time spent starting KDC: real 1.106s user 0.005s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781986536687152-17252-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781986536687152-17252-0/krb5kdc/test-user.keytab.
Jun 20 20:21:01 dist-test-slave-2hpg krb5kdc[5278](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986861, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T20:21:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T20:21:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781986536687152-17252-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781986536687152-17252-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 20:21:01.894418 17252 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
/tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781986536687152-17252-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781986536687152-17252-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781986536687152-17252-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781986536687152-17252-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38813
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781986536687152-17252-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38813
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781986536687152-17252-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781986536687152-17252-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781986536687152-17252-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:21:02.227716 5291 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:21:02.227994 5291 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:21:02.228178 5291 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:21:02.235468 5291 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:21:02.235596 5291 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:21:02.235711 5291 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:21:02.235827 5291 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:21:02.235911 5291 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:21:02.245749 5291 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781986536687152-17252-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781986536687152-17252-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38813
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781986536687152-17252-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38813
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781986536687152-17252-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781986536687152-17252-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:21:02.247105 5291 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:21:02.248414 5291 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:21:02.260143 5297 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:21:02.260228 5296 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:21:02.260201 5299 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:21:02.260993 5291 server_base.cc:1034] running on GCE node
I20260620 20:21:02.261745 5291 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:21:02.263149 5291 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:21:02.264346 5291 hybrid_clock.cc:648] HybridClock initialized: now 1781986862264285 us; error 51 us; skew 500 ppm
I20260620 20:21:02.264683 5291 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:21:02 dist-test-slave-2hpg krb5kdc[5278](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986862, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:21:02.278163 5291 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 20:21:02.279870 5291 webserver.cc:460] Webserver started at http://127.0.0.1:42655/ using document root <none> and password file <none>
I20260620 20:21:02.280594 5291 fs_manager.cc:359] Metadata directory not provided
I20260620 20:21:02.280709 5291 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:21:02.281005 5291 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:21:02.284371 5291 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781986536687152-17252-0/minicluster-data/master-0/data/instance:
uuid: "435542c425144191a0d811997bb503a1"
format_stamp: "Formatted at 2026-06-20 20:21:02 on dist-test-slave-2hpg"
server_key: "e550461034ea23add2bcc0635b1c2fb7"
server_key_iv: "c0008bcfdd68c15e22cc7a8c8081e463"
server_key_version: "encryptionkey@0"
I20260620 20:21:02.285173 5291 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781986536687152-17252-0/minicluster-data/master-0/wal/instance:
uuid: "435542c425144191a0d811997bb503a1"
format_stamp: "Formatted at 2026-06-20 20:21:02 on dist-test-slave-2hpg"
server_key: "e550461034ea23add2bcc0635b1c2fb7"
server_key_iv: "c0008bcfdd68c15e22cc7a8c8081e463"
server_key_version: "encryptionkey@0"
I20260620 20:21:02.291364 5291 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260620 20:21:02.295012 5306 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:21:02.296562 5291 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 20:21:02.296772 5291 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781986536687152-17252-0/minicluster-data/master-0/data,/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781986536687152-17252-0/minicluster-data/master-0/wal
uuid: "435542c425144191a0d811997bb503a1"
format_stamp: "Formatted at 2026-06-20 20:21:02 on dist-test-slave-2hpg"
server_key: "e550461034ea23add2bcc0635b1c2fb7"
server_key_iv: "c0008bcfdd68c15e22cc7a8c8081e463"
server_key_version: "encryptionkey@0"
I20260620 20:21:02.296993 5291 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781986536687152-17252-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781986536687152-17252-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781986536687152-17252-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:21:02.321527 5291 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:21:02.326211 5291 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:21:02.326511 5291 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:21:02.345937 5291 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38813
I20260620 20:21:02.345993 5357 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38813 every 8 connection(s)
I20260620 20:21:02.347347 5291 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781986536687152-17252-0/minicluster-data/master-0/data/info.pb
I20260620 20:21:02.348438 17252 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu as pid 5291
I20260620 20:21:02.348672 17252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781986536687152-17252-0/minicluster-data/master-0/wal/instance
I20260620 20:21:02.349113 17252 external_mini_cluster.cc:1428] Setting key cf7a6c3a1ec00987f896ea497136059d
I20260620 20:21:02.351876 5358 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:21:02.364256 5361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:02.350957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36409 (local address 127.0.0.1:38813)
0620 20:21:02.351707 (+ 750us) server_negotiation.cc:206] Beginning negotiation
0620 20:21:02.351721 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 20:21:02.351770 (+ 49us) server_negotiation.cc:407] Connection header received
0620 20:21:02.354180 (+ 2410us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:21:02.354205 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:21:02.354700 (+ 495us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:21:02.355155 (+ 455us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:21:02.355705 (+ 550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:02.357696 (+ 1991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:02.359046 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:02.360069 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:02.360249 (+ 180us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:21:02.361454 (+ 1205us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:21:02.361465 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:21:02.361481 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:21:02.362255 (+ 774us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:21:02.362349 (+ 94us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:21:02.362385 (+ 36us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:21:02.362561 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0620 20:21:02.362812 (+ 251us) server_negotiation.cc:299] Negotiation successful
0620 20:21:02.362998 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":525,"thread_start_us":304,"threads_started":1}
I20260620 20:21:02.365561 5358 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 435542c425144191a0d811997bb503a1: Bootstrap starting.
I20260620 20:21:02.368075 17252 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 20:21:02.368772 5358 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 435542c425144191a0d811997bb503a1: Neither blocks nor log segments found. Creating new log.
I20260620 20:21:02.369721 5358 log.cc:824] T 00000000000000000000000000000000 P 435542c425144191a0d811997bb503a1: Log is configured to *not* fsync() on all Append() calls
I20260620 20:21:02.372993 5358 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 435542c425144191a0d811997bb503a1: No bootstrap required, opened a new log
I20260620 20:21:02.379900 5361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:02.372517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33703 (local address 127.0.0.1:38813)
0620 20:21:02.372867 (+ 350us) server_negotiation.cc:206] Beginning negotiation
0620 20:21:02.372874 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:21:02.373203 (+ 329us) server_negotiation.cc:407] Connection header received
0620 20:21:02.373572 (+ 369us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:21:02.373579 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:21:02.373657 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:21:02.373730 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:21:02.374592 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:02.375812 (+ 1220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:02.377093 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:02.377797 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:02.377990 (+ 193us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:21:02.379075 (+ 1085us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:21:02.379082 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:21:02.379087 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:21:02.379107 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:21:02.379174 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:21:02.379196 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:21:02.379329 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0620 20:21:02.379475 (+ 146us) server_negotiation.cc:299] Negotiation successful
0620 20:21:02.379541 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260620 20:21:02.382566 5358 raft_consensus.cc:348] T 00000000000000000000000000000000 P 435542c425144191a0d811997bb503a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "435542c425144191a0d811997bb503a1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38813 } }
I20260620 20:21:02.382898 5358 raft_consensus.cc:374] T 00000000000000000000000000000000 P 435542c425144191a0d811997bb503a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:21:02.382974 5358 raft_consensus.cc:729] T 00000000000000000000000000000000 P 435542c425144191a0d811997bb503a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 435542c425144191a0d811997bb503a1, State: Initialized, Role: FOLLOWER
I20260620 20:21:02.383455 5358 consensus_queue.cc:260] T 00000000000000000000000000000000 P 435542c425144191a0d811997bb503a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "435542c425144191a0d811997bb503a1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38813 } }
I20260620 20:21:02.383641 5358 raft_consensus.cc:388] T 00000000000000000000000000000000 P 435542c425144191a0d811997bb503a1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 20:21:02.383735 5358 raft_consensus.cc:482] T 00000000000000000000000000000000 P 435542c425144191a0d811997bb503a1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 20:21:02.383854 5358 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 435542c425144191a0d811997bb503a1 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:21:02.386021 5358 raft_consensus.cc:504] T 00000000000000000000000000000000 P 435542c425144191a0d811997bb503a1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "435542c425144191a0d811997bb503a1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38813 } }
I20260620 20:21:02.386521 5358 leader_election.cc:302] T 00000000000000000000000000000000 P 435542c425144191a0d811997bb503a1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 435542c425144191a0d811997bb503a1; no voters:
I20260620 20:21:02.387068 5358 leader_election.cc:288] T 00000000000000000000000000000000 P 435542c425144191a0d811997bb503a1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 20:21:02.387236 5368 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 435542c425144191a0d811997bb503a1 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:21:02.388006 5368 raft_consensus.cc:686] T 00000000000000000000000000000000 P 435542c425144191a0d811997bb503a1 [term 1 LEADER]: Becoming Leader. State: Replica: 435542c425144191a0d811997bb503a1, State: Running, Role: LEADER
I20260620 20:21:02.388525 5368 consensus_queue.cc:237] T 00000000000000000000000000000000 P 435542c425144191a0d811997bb503a1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "435542c425144191a0d811997bb503a1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38813 } }
I20260620 20:21:02.389156 5358 sys_catalog.cc:564] T 00000000000000000000000000000000 P 435542c425144191a0d811997bb503a1 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:21:02.391348 5369 sys_catalog.cc:455] T 00000000000000000000000000000000 P 435542c425144191a0d811997bb503a1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "435542c425144191a0d811997bb503a1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "435542c425144191a0d811997bb503a1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38813 } } }
I20260620 20:21:02.391680 5369 sys_catalog.cc:458] T 00000000000000000000000000000000 P 435542c425144191a0d811997bb503a1 [sys.catalog]: This master's current role is: LEADER
I20260620 20:21:02.392104 5370 sys_catalog.cc:455] T 00000000000000000000000000000000 P 435542c425144191a0d811997bb503a1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 435542c425144191a0d811997bb503a1. Latest consensus state: current_term: 1 leader_uuid: "435542c425144191a0d811997bb503a1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "435542c425144191a0d811997bb503a1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38813 } } }
I20260620 20:21:02.392385 5370 sys_catalog.cc:458] T 00000000000000000000000000000000 P 435542c425144191a0d811997bb503a1 [sys.catalog]: This master's current role is: LEADER
I20260620 20:21:02.394613 5375 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 20:21:02.399891 5375 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 20:21:02.409170 5375 catalog_manager.cc:1347] Generated new cluster ID: 28e7a9a5f10b4ab38bf69879aed90aad
I20260620 20:21:02.409309 5375 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 20:21:02.434450 5375 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 20:21:02.436393 5375 catalog_manager.cc:1504] Loading token signing keys...
I20260620 20:21:02.452024 5375 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 435542c425144191a0d811997bb503a1: Generated new TSK 0
I20260620 20:21:02.452900 5375 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 20:21:02.468849 17252 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu pid 5291
W20260620 20:21:02.470077 5353 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:36409
W20260620 20:21:02.470889 17252 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu pid 5291: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 20:21:02.471095 17252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu with pid 5291
W20260620 20:21:02.488183 5276 connection.cc:535] client connection to 127.0.0.1:38813 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:38813): Connection reset by peer (error 104)
2026-06-20T20:21:02Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1963 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781986536687152-17252-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 20:21:02 dist-test-slave-2hpg krb5kdc[5389](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 20:21:02 dist-test-slave-2hpg krb5kdc[5389](info): set up 2 sockets
Jun 20 20:21:02 dist-test-slave-2hpg krb5kdc[5389](info): commencing operation
krb5kdc: starting...
W20260620 20:21:03.570853 17252 mini_kdc.cc:121] Time spent starting KDC: real 1.064s user 0.003s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781986536687152-17252-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781986536687152-17252-0/krb5kdc/test-user.keytab.
Jun 20 20:21:03 dist-test-slave-2hpg krb5kdc[5389](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986863, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T20:21:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T20:21:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781986536687152-17252-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781986536687152-17252-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 20:21:03.822026 17252 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
/tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781986536687152-17252-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781986536687152-17252-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781986536687152-17252-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781986536687152-17252-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39333
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781986536687152-17252-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44271
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39333
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781986536687152-17252-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781986536687152-17252-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781986536687152-17252-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:21:04.156713 5402 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:21:04.157055 5402 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:21:04.157232 5402 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:21:04.164686 5402 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:21:04.164840 5402 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:21:04.164948 5402 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:21:04.165052 5402 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:21:04.165143 5402 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:21:04.175341 5402 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44271
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781986536687152-17252-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781986536687152-17252-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39333
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781986536687152-17252-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39333
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781986536687152-17252-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781986536687152-17252-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:21:04.176705 5402 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:21:04.178121 5402 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:21:04.190236 5407 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:21:04.190294 5410 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:21:04.190351 5408 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:21:04.191674 5402 server_base.cc:1034] running on GCE node
I20260620 20:21:04.192490 5402 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:21:04.194070 5402 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:21:04.195314 5402 hybrid_clock.cc:648] HybridClock initialized: now 1781986864195272 us; error 36 us; skew 500 ppm
I20260620 20:21:04.195669 5402 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:21:04 dist-test-slave-2hpg krb5kdc[5389](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986864, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:21:04.198858 5402 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 20:21:04.200311 5402 webserver.cc:460] Webserver started at http://127.0.0.1:44881/ using document root <none> and password file <none>
I20260620 20:21:04.200949 5402 fs_manager.cc:359] Metadata directory not provided
I20260620 20:21:04.201090 5402 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:21:04.201447 5402 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:21:04.204465 5402 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781986536687152-17252-0/minicluster-data/master-0/data/instance:
uuid: "9489de27a039456b9b705bf60c3eefe6"
format_stamp: "Formatted at 2026-06-20 20:21:04 on dist-test-slave-2hpg"
server_key: "2393e186d332a7fa534866da50b37049"
server_key_iv: "ef1f4838dac789bce25e5f26c5112c3f"
server_key_version: "encryptionkey@0"
I20260620 20:21:04.205246 5402 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781986536687152-17252-0/minicluster-data/master-0/wal/instance:
uuid: "9489de27a039456b9b705bf60c3eefe6"
format_stamp: "Formatted at 2026-06-20 20:21:04 on dist-test-slave-2hpg"
server_key: "2393e186d332a7fa534866da50b37049"
server_key_iv: "ef1f4838dac789bce25e5f26c5112c3f"
server_key_version: "encryptionkey@0"
I20260620 20:21:04.210268 5402 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260620 20:21:04.213629 5417 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:21:04.214946 5402 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260620 20:21:04.215121 5402 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781986536687152-17252-0/minicluster-data/master-0/data,/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781986536687152-17252-0/minicluster-data/master-0/wal
uuid: "9489de27a039456b9b705bf60c3eefe6"
format_stamp: "Formatted at 2026-06-20 20:21:04 on dist-test-slave-2hpg"
server_key: "2393e186d332a7fa534866da50b37049"
server_key_iv: "ef1f4838dac789bce25e5f26c5112c3f"
server_key_version: "encryptionkey@0"
I20260620 20:21:04.215307 5402 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781986536687152-17252-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781986536687152-17252-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781986536687152-17252-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:21:04.221243 5402 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:21:04.225495 5402 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:21:04.225791 5402 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:21:04.244858 5402 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39333
I20260620 20:21:04.244884 5468 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39333 every 8 connection(s)
I20260620 20:21:04.246313 5402 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781986536687152-17252-0/minicluster-data/master-0/data/info.pb
I20260620 20:21:04.250171 5469 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:21:04.255170 17252 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu as pid 5402
I20260620 20:21:04.255353 17252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781986536687152-17252-0/minicluster-data/master-0/wal/instance
I20260620 20:21:04.255765 17252 external_mini_cluster.cc:1428] Setting key 09b9cbacf9188dd079624cf07a995a63
I20260620 20:21:04.262144 5469 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9489de27a039456b9b705bf60c3eefe6: Bootstrap starting.
I20260620 20:21:04.264638 5472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:04.257490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36141 (local address 127.0.0.1:39333)
0620 20:21:04.258368 (+ 878us) server_negotiation.cc:206] Beginning negotiation
0620 20:21:04.258383 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 20:21:04.258434 (+ 51us) server_negotiation.cc:407] Connection header received
0620 20:21:04.260879 (+ 2445us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:21:04.260905 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:21:04.261341 (+ 436us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:21:04.261769 (+ 428us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:21:04.262747 (+ 978us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:21:04.262755 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:21:04.262771 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:21:04.262786 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:21:04.262848 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:21:04.262872 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:21:04.263083 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0620 20:21:04.263204 (+ 121us) server_negotiation.cc:299] Negotiation successful
0620 20:21:04.263373 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":597,"thread_start_us":368,"threads_started":1}
I20260620 20:21:04.265800 5469 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9489de27a039456b9b705bf60c3eefe6: Neither blocks nor log segments found. Creating new log.
I20260620 20:21:04.266939 5469 log.cc:824] T 00000000000000000000000000000000 P 9489de27a039456b9b705bf60c3eefe6: Log is configured to *not* fsync() on all Append() calls
I20260620 20:21:04.268251 17252 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 20:21:04.270124 5469 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9489de27a039456b9b705bf60c3eefe6: No bootstrap required, opened a new log
I20260620 20:21:04.276180 5472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:04.273476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36567 (local address 127.0.0.1:39333)
0620 20:21:04.273659 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0620 20:21:04.273665 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:21:04.274070 (+ 405us) server_negotiation.cc:407] Connection header received
0620 20:21:04.274180 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:21:04.274190 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:21:04.274251 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:21:04.274364 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:21:04.275493 (+ 1129us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:21:04.275499 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:21:04.275504 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:21:04.275524 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:21:04.275582 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:21:04.275602 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:21:04.275723 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0620 20:21:04.275832 (+ 109us) server_negotiation.cc:299] Negotiation successful
0620 20:21:04.275889 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260620 20:21:04.279719 5469 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9489de27a039456b9b705bf60c3eefe6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9489de27a039456b9b705bf60c3eefe6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39333 } }
I20260620 20:21:04.280073 5469 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9489de27a039456b9b705bf60c3eefe6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:21:04.280148 5469 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9489de27a039456b9b705bf60c3eefe6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9489de27a039456b9b705bf60c3eefe6, State: Initialized, Role: FOLLOWER
I20260620 20:21:04.280629 5469 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9489de27a039456b9b705bf60c3eefe6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9489de27a039456b9b705bf60c3eefe6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39333 } }
I20260620 20:21:04.280838 5469 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9489de27a039456b9b705bf60c3eefe6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 20:21:04.280943 5469 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9489de27a039456b9b705bf60c3eefe6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 20:21:04.281159 5469 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9489de27a039456b9b705bf60c3eefe6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:21:04.283699 5469 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9489de27a039456b9b705bf60c3eefe6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9489de27a039456b9b705bf60c3eefe6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39333 } }
I20260620 20:21:04.284190 5469 leader_election.cc:302] T 00000000000000000000000000000000 P 9489de27a039456b9b705bf60c3eefe6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9489de27a039456b9b705bf60c3eefe6; no voters:
I20260620 20:21:04.284747 5469 leader_election.cc:288] T 00000000000000000000000000000000 P 9489de27a039456b9b705bf60c3eefe6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 20:21:04.284924 5479 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9489de27a039456b9b705bf60c3eefe6 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:21:04.285647 5479 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9489de27a039456b9b705bf60c3eefe6 [term 1 LEADER]: Becoming Leader. State: Replica: 9489de27a039456b9b705bf60c3eefe6, State: Running, Role: LEADER
I20260620 20:21:04.286194 5479 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9489de27a039456b9b705bf60c3eefe6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9489de27a039456b9b705bf60c3eefe6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39333 } }
I20260620 20:21:04.286947 5469 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9489de27a039456b9b705bf60c3eefe6 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:21:04.290716 5481 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9489de27a039456b9b705bf60c3eefe6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9489de27a039456b9b705bf60c3eefe6. Latest consensus state: current_term: 1 leader_uuid: "9489de27a039456b9b705bf60c3eefe6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9489de27a039456b9b705bf60c3eefe6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39333 } } }
I20260620 20:21:04.290741 5480 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9489de27a039456b9b705bf60c3eefe6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9489de27a039456b9b705bf60c3eefe6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9489de27a039456b9b705bf60c3eefe6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39333 } } }
I20260620 20:21:04.291177 5480 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9489de27a039456b9b705bf60c3eefe6 [sys.catalog]: This master's current role is: LEADER
I20260620 20:21:04.291173 5481 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9489de27a039456b9b705bf60c3eefe6 [sys.catalog]: This master's current role is: LEADER
I20260620 20:21:04.294639 5486 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 20:21:04.300102 5486 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 20:21:04.309423 5486 catalog_manager.cc:1347] Generated new cluster ID: c144ea6b010b42908f7371492825b083
I20260620 20:21:04.309545 5486 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 20:21:04.353361 5486 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 20:21:04.353804 5486 catalog_manager.cc:1504] Loading token signing keys...
I20260620 20:21:04.361151 5486 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9489de27a039456b9b705bf60c3eefe6: Generated new TSK 0
I20260620 20:21:04.362037 5486 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 20:21:04.373438 17252 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu pid 5402
W20260620 20:21:04.374516 5464 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:36141
W20260620 20:21:04.375242 17252 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu pid 5402: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 20:21:04.375381 17252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu with pid 5402
2026-06-20T20:21:04Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1903 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781986536687152-17252-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 20:21:04 dist-test-slave-2hpg krb5kdc[5500](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 20:21:04 dist-test-slave-2hpg krb5kdc[5500](info): set up 2 sockets
Jun 20 20:21:04 dist-test-slave-2hpg krb5kdc[5500](info): commencing operation
krb5kdc: starting...
W20260620 20:21:05.494658 17252 mini_kdc.cc:121] Time spent starting KDC: real 1.084s user 0.003s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781986536687152-17252-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781986536687152-17252-0/krb5kdc/test-user.keytab.
Jun 20 20:21:05 dist-test-slave-2hpg krb5kdc[5500](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986865, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T20:21:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T20:21:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781986536687152-17252-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781986536687152-17252-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 20:21:05.746027 17252 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
/tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781986536687152-17252-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781986536687152-17252-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781986536687152-17252-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781986536687152-17252-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38815
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781986536687152-17252-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39927
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38815
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781986536687152-17252-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781986536687152-17252-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781986536687152-17252-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:21:06.081449 5513 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:21:06.081720 5513 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:21:06.081811 5513 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:21:06.088927 5513 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:21:06.089025 5513 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:21:06.089071 5513 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:21:06.089130 5513 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:21:06.089174 5513 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:21:06.098946 5513 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39927
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781986536687152-17252-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781986536687152-17252-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38815
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781986536687152-17252-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38815
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781986536687152-17252-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781986536687152-17252-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:21:06.100148 5513 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:21:06.101418 5513 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:21:06.113042 5518 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:21:06.113130 5519 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:21:06.113087 5521 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:21:06.113982 5513 server_base.cc:1034] running on GCE node
I20260620 20:21:06.114698 5513 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:21:06.116036 5513 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:21:06.117298 5513 hybrid_clock.cc:648] HybridClock initialized: now 1781986866117263 us; error 54 us; skew 500 ppm
I20260620 20:21:06.117686 5513 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:21:06 dist-test-slave-2hpg krb5kdc[5500](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986866, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:21:06.120994 5513 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 20:21:06.122514 5513 webserver.cc:460] Webserver started at http://127.0.0.1:44395/ using document root <none> and password file <none>
I20260620 20:21:06.123190 5513 fs_manager.cc:359] Metadata directory not provided
I20260620 20:21:06.123322 5513 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:21:06.123653 5513 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:21:06.126708 5513 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781986536687152-17252-0/minicluster-data/master-0/data/instance:
uuid: "99f91cece70c442e8996cee6add99366"
format_stamp: "Formatted at 2026-06-20 20:21:06 on dist-test-slave-2hpg"
server_key: "b042f1961b7496af0529a69a2829425d"
server_key_iv: "7a4334189f74fc49f8a3521ea278997b"
server_key_version: "encryptionkey@0"
I20260620 20:21:06.127489 5513 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781986536687152-17252-0/minicluster-data/master-0/wal/instance:
uuid: "99f91cece70c442e8996cee6add99366"
format_stamp: "Formatted at 2026-06-20 20:21:06 on dist-test-slave-2hpg"
server_key: "b042f1961b7496af0529a69a2829425d"
server_key_iv: "7a4334189f74fc49f8a3521ea278997b"
server_key_version: "encryptionkey@0"
I20260620 20:21:06.132637 5513 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260620 20:21:06.136363 5528 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:21:06.137732 5513 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 20:21:06.137967 5513 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781986536687152-17252-0/minicluster-data/master-0/data,/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781986536687152-17252-0/minicluster-data/master-0/wal
uuid: "99f91cece70c442e8996cee6add99366"
format_stamp: "Formatted at 2026-06-20 20:21:06 on dist-test-slave-2hpg"
server_key: "b042f1961b7496af0529a69a2829425d"
server_key_iv: "7a4334189f74fc49f8a3521ea278997b"
server_key_version: "encryptionkey@0"
I20260620 20:21:06.138185 5513 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781986536687152-17252-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781986536687152-17252-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781986536687152-17252-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:21:06.179536 5513 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:21:06.184249 5513 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:21:06.184597 5513 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:21:06.205135 5513 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38815
I20260620 20:21:06.205160 5579 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38815 every 8 connection(s)
I20260620 20:21:06.206634 5513 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781986536687152-17252-0/minicluster-data/master-0/data/info.pb
I20260620 20:21:06.210022 17252 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu as pid 5513
I20260620 20:21:06.210246 17252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781986536687152-17252-0/minicluster-data/master-0/wal/instance
I20260620 20:21:06.210548 5580 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:21:06.210767 17252 external_mini_cluster.cc:1428] Setting key 9a68dbbc315ebc852f038cb002036877
I20260620 20:21:06.224812 5580 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 99f91cece70c442e8996cee6add99366: Bootstrap starting.
Jun 20 20:21:06 dist-test-slave-2hpg krb5kdc[5500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986865, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 20:21:06.228123 5580 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 99f91cece70c442e8996cee6add99366: Neither blocks nor log segments found. Creating new log.
I20260620 20:21:06.229516 5580 log.cc:824] T 00000000000000000000000000000000 P 99f91cece70c442e8996cee6add99366: Log is configured to *not* fsync() on all Append() calls
I20260620 20:21:06.232930 5580 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 99f91cece70c442e8996cee6add99366: No bootstrap required, opened a new log
I20260620 20:21:06.237138 5583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:06.212831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33655 (local address 127.0.0.1:38815)
0620 20:21:06.213846 (+ 1015us) server_negotiation.cc:206] Beginning negotiation
0620 20:21:06.213885 (+ 39us) server_negotiation.cc:399] Waiting for connection header
0620 20:21:06.214271 (+ 386us) server_negotiation.cc:407] Connection header received
0620 20:21:06.217034 (+ 2763us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:21:06.217063 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:21:06.217591 (+ 528us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:21:06.218079 (+ 488us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:21:06.219393 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:06.221422 (+ 2029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:06.222774 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:06.223802 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:06.223976 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:21:06.228265 (+ 4289us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:21:06.228276 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:21:06.228298 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:21:06.228356 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:21:06.232430 (+ 4074us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:21:06.233127 (+ 697us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:21:06.233135 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:21:06.233143 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:21:06.233346 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:21:06.233857 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:21:06.233897 (+ 40us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:21:06.233900 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:21:06.234583 (+ 683us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:21:06.234900 (+ 317us) server_negotiation.cc:1035] Waiting for connection context
0620 20:21:06.235300 (+ 400us) server_negotiation.cc:299] Negotiation successful
0620 20:21:06.235767 (+ 467us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":735,"thread_start_us":443,"threads_started":1}
I20260620 20:21:06.241031 17252 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 20:21:06.243084 5580 raft_consensus.cc:348] T 00000000000000000000000000000000 P 99f91cece70c442e8996cee6add99366 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99f91cece70c442e8996cee6add99366" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38815 } }
I20260620 20:21:06.243448 5580 raft_consensus.cc:374] T 00000000000000000000000000000000 P 99f91cece70c442e8996cee6add99366 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:21:06.243570 5580 raft_consensus.cc:729] T 00000000000000000000000000000000 P 99f91cece70c442e8996cee6add99366 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99f91cece70c442e8996cee6add99366, State: Initialized, Role: FOLLOWER
I20260620 20:21:06.244223 5580 consensus_queue.cc:260] T 00000000000000000000000000000000 P 99f91cece70c442e8996cee6add99366 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99f91cece70c442e8996cee6add99366" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38815 } }
I20260620 20:21:06.244598 5580 raft_consensus.cc:388] T 00000000000000000000000000000000 P 99f91cece70c442e8996cee6add99366 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 20:21:06.244750 5580 raft_consensus.cc:482] T 00000000000000000000000000000000 P 99f91cece70c442e8996cee6add99366 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 20:21:06.244930 5580 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 99f91cece70c442e8996cee6add99366 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:21:06.247677 5580 raft_consensus.cc:504] T 00000000000000000000000000000000 P 99f91cece70c442e8996cee6add99366 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99f91cece70c442e8996cee6add99366" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38815 } }
I20260620 20:21:06.248202 5580 leader_election.cc:302] T 00000000000000000000000000000000 P 99f91cece70c442e8996cee6add99366 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 99f91cece70c442e8996cee6add99366; no voters:
I20260620 20:21:06.248894 5580 leader_election.cc:288] T 00000000000000000000000000000000 P 99f91cece70c442e8996cee6add99366 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 20:21:06.248999 5590 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 99f91cece70c442e8996cee6add99366 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:21:06.251531 5580 sys_catalog.cc:564] T 00000000000000000000000000000000 P 99f91cece70c442e8996cee6add99366 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:21:06.256076 5590 raft_consensus.cc:686] T 00000000000000000000000000000000 P 99f91cece70c442e8996cee6add99366 [term 1 LEADER]: Becoming Leader. State: Replica: 99f91cece70c442e8996cee6add99366, State: Running, Role: LEADER
I20260620 20:21:06.256873 5590 consensus_queue.cc:237] T 00000000000000000000000000000000 P 99f91cece70c442e8996cee6add99366 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99f91cece70c442e8996cee6add99366" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38815 } }
I20260620 20:21:06.260298 5591 sys_catalog.cc:455] T 00000000000000000000000000000000 P 99f91cece70c442e8996cee6add99366 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "99f91cece70c442e8996cee6add99366" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99f91cece70c442e8996cee6add99366" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38815 } } }
I20260620 20:21:06.260780 5591 sys_catalog.cc:458] T 00000000000000000000000000000000 P 99f91cece70c442e8996cee6add99366 [sys.catalog]: This master's current role is: LEADER
I20260620 20:21:06.261107 5583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:06.245679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40231 (local address 127.0.0.1:38815)
0620 20:21:06.245842 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0620 20:21:06.245846 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:21:06.246363 (+ 517us) server_negotiation.cc:407] Connection header received
0620 20:21:06.246595 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:21:06.246602 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:21:06.246676 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:21:06.246793 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:21:06.248287 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:06.249171 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:06.250745 (+ 1574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:06.251332 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:06.251468 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:21:06.254234 (+ 2766us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:21:06.254240 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:21:06.254244 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:21:06.254280 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:21:06.257305 (+ 3025us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:21:06.258123 (+ 818us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:21:06.258129 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:21:06.258132 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:21:06.258292 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:21:06.258688 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:21:06.258693 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:21:06.258698 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:21:06.259150 (+ 452us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:21:06.259416 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0620 20:21:06.260523 (+ 1107us) server_negotiation.cc:299] Negotiation successful
0620 20:21:06.260746 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 20:21:06.261747 5598 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 20:21:06.261786 5592 sys_catalog.cc:455] T 00000000000000000000000000000000 P 99f91cece70c442e8996cee6add99366 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 99f91cece70c442e8996cee6add99366. Latest consensus state: current_term: 1 leader_uuid: "99f91cece70c442e8996cee6add99366" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99f91cece70c442e8996cee6add99366" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38815 } } }
I20260620 20:21:06.262135 5592 sys_catalog.cc:458] T 00000000000000000000000000000000 P 99f91cece70c442e8996cee6add99366 [sys.catalog]: This master's current role is: LEADER
I20260620 20:21:06.269603 5598 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 20:21:06.278743 5598 catalog_manager.cc:1347] Generated new cluster ID: 298f775594e24432a12b6fe0e8b1fa38
I20260620 20:21:06.278880 5598 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 20:21:06.315639 5598 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 20:21:06.317644 5598 catalog_manager.cc:1504] Loading token signing keys...
I20260620 20:21:06.332188 5598 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 99f91cece70c442e8996cee6add99366: Generated new TSK 0
I20260620 20:21:06.333241 5598 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 20:21:06.344739 17252 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu pid 5513
I20260620 20:21:06.345671 5575 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:21:06.440160 17252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu with pid 5513
2026-06-20T20:21:06Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (2062 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781986536687152-17252-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 20:21:06 dist-test-slave-2hpg krb5kdc[5613](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 20:21:06 dist-test-slave-2hpg krb5kdc[5613](info): set up 2 sockets
Jun 20 20:21:06 dist-test-slave-2hpg krb5kdc[5613](info): commencing operation
krb5kdc: starting...
W20260620 20:21:07.546518 17252 mini_kdc.cc:121] Time spent starting KDC: real 1.073s user 0.000s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781986536687152-17252-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781986536687152-17252-0/krb5kdc/test-user.keytab.
Jun 20 20:21:07 dist-test-slave-2hpg krb5kdc[5613](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986867, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T20:21:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T20:21:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781986536687152-17252-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781986536687152-17252-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 20:21:07.784839 17252 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
/tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781986536687152-17252-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781986536687152-17252-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781986536687152-17252-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781986536687152-17252-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40215
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781986536687152-17252-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42927
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40215
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781986536687152-17252-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781986536687152-17252-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781986536687152-17252-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:21:08.116495 5626 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:21:08.116835 5626 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:21:08.117036 5626 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:21:08.124260 5626 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:21:08.124372 5626 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:21:08.124418 5626 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:21:08.124473 5626 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:21:08.124526 5626 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:21:08.134267 5626 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42927
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781986536687152-17252-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781986536687152-17252-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40215
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781986536687152-17252-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40215
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781986536687152-17252-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781986536687152-17252-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:21:08.135465 5626 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:21:08.136734 5626 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:21:08.148651 5632 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:21:08.148783 5631 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:21:08.148869 5634 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:21:08.149669 5626 server_base.cc:1034] running on GCE node
I20260620 20:21:08.150485 5626 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:21:08.151854 5626 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:21:08.153097 5626 hybrid_clock.cc:648] HybridClock initialized: now 1781986868153040 us; error 46 us; skew 500 ppm
I20260620 20:21:08.153452 5626 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:21:08 dist-test-slave-2hpg krb5kdc[5613](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986868, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:21:08.160475 5626 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 20:21:08.162036 5626 webserver.cc:460] Webserver started at http://127.0.0.1:37227/ using document root <none> and password file <none>
I20260620 20:21:08.162676 5626 fs_manager.cc:359] Metadata directory not provided
I20260620 20:21:08.162781 5626 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:21:08.163038 5626 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:21:08.166018 5626 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781986536687152-17252-0/minicluster-data/master-0/data/instance:
uuid: "670fdb5994a74b19a3517a3d052e6b67"
format_stamp: "Formatted at 2026-06-20 20:21:08 on dist-test-slave-2hpg"
server_key: "4fd2980122885b3d8744e76db5bc6487"
server_key_iv: "84d5232e35fb53ab9fdaec61c25edd92"
server_key_version: "encryptionkey@0"
I20260620 20:21:08.166733 5626 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781986536687152-17252-0/minicluster-data/master-0/wal/instance:
uuid: "670fdb5994a74b19a3517a3d052e6b67"
format_stamp: "Formatted at 2026-06-20 20:21:08 on dist-test-slave-2hpg"
server_key: "4fd2980122885b3d8744e76db5bc6487"
server_key_iv: "84d5232e35fb53ab9fdaec61c25edd92"
server_key_version: "encryptionkey@0"
I20260620 20:21:08.171738 5626 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260620 20:21:08.175099 5641 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:21:08.176375 5626 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 20:21:08.176509 5626 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781986536687152-17252-0/minicluster-data/master-0/data,/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781986536687152-17252-0/minicluster-data/master-0/wal
uuid: "670fdb5994a74b19a3517a3d052e6b67"
format_stamp: "Formatted at 2026-06-20 20:21:08 on dist-test-slave-2hpg"
server_key: "4fd2980122885b3d8744e76db5bc6487"
server_key_iv: "84d5232e35fb53ab9fdaec61c25edd92"
server_key_version: "encryptionkey@0"
I20260620 20:21:08.176666 5626 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781986536687152-17252-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781986536687152-17252-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781986536687152-17252-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:21:08.192837 5626 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:21:08.196660 5626 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:21:08.196887 5626 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:21:08.215936 5626 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40215
I20260620 20:21:08.215957 5692 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40215 every 8 connection(s)
I20260620 20:21:08.217281 5626 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781986536687152-17252-0/minicluster-data/master-0/data/info.pb
I20260620 20:21:08.218950 17252 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu as pid 5626
I20260620 20:21:08.219166 17252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781986536687152-17252-0/minicluster-data/master-0/wal/instance
I20260620 20:21:08.219756 17252 external_mini_cluster.cc:1428] Setting key 65f8b22b08a27117ad6ecd479f964ead
I20260620 20:21:08.221994 5693 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:21:08.232690 5696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:08.221617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48737 (local address 127.0.0.1:40215)
0620 20:21:08.222459 (+ 842us) server_negotiation.cc:206] Beginning negotiation
0620 20:21:08.222471 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 20:21:08.222519 (+ 48us) server_negotiation.cc:407] Connection header received
0620 20:21:08.224199 (+ 1680us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:21:08.224220 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:21:08.224606 (+ 386us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:21:08.224998 (+ 392us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:21:08.225547 (+ 549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:08.226929 (+ 1382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:08.228247 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:08.228903 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:08.229092 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:21:08.230416 (+ 1324us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:21:08.230431 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:21:08.230443 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:21:08.231044 (+ 601us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:21:08.231121 (+ 77us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:21:08.231147 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:21:08.231307 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0620 20:21:08.231538 (+ 231us) server_negotiation.cc:299] Negotiation successful
0620 20:21:08.231788 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":642,"thread_start_us":419,"threads_started":1}
I20260620 20:21:08.233139 5693 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 670fdb5994a74b19a3517a3d052e6b67: Bootstrap starting.
I20260620 20:21:08.236229 17252 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 20:21:08.236411 5693 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 670fdb5994a74b19a3517a3d052e6b67: Neither blocks nor log segments found. Creating new log.
I20260620 20:21:08.237394 5693 log.cc:824] T 00000000000000000000000000000000 P 670fdb5994a74b19a3517a3d052e6b67: Log is configured to *not* fsync() on all Append() calls
I20260620 20:21:08.240335 5693 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 670fdb5994a74b19a3517a3d052e6b67: No bootstrap required, opened a new log
I20260620 20:21:08.247786 5696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:08.240884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50591 (local address 127.0.0.1:40215)
0620 20:21:08.241079 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0620 20:21:08.241085 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:21:08.241466 (+ 381us) server_negotiation.cc:407] Connection header received
0620 20:21:08.241625 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:21:08.241630 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:21:08.241700 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:21:08.241808 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:21:08.242312 (+ 504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:08.243457 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:08.244788 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:08.245572 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:08.245715 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:21:08.246990 (+ 1275us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:21:08.246997 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:21:08.247001 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:21:08.247023 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:21:08.247076 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:21:08.247096 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:21:08.247197 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0620 20:21:08.247361 (+ 164us) server_negotiation.cc:299] Negotiation successful
0620 20:21:08.247463 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 20:21:08.250144 5693 raft_consensus.cc:348] T 00000000000000000000000000000000 P 670fdb5994a74b19a3517a3d052e6b67 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "670fdb5994a74b19a3517a3d052e6b67" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40215 } }
I20260620 20:21:08.250514 5693 raft_consensus.cc:374] T 00000000000000000000000000000000 P 670fdb5994a74b19a3517a3d052e6b67 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:21:08.250589 5693 raft_consensus.cc:729] T 00000000000000000000000000000000 P 670fdb5994a74b19a3517a3d052e6b67 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 670fdb5994a74b19a3517a3d052e6b67, State: Initialized, Role: FOLLOWER
I20260620 20:21:08.251143 5693 consensus_queue.cc:260] T 00000000000000000000000000000000 P 670fdb5994a74b19a3517a3d052e6b67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "670fdb5994a74b19a3517a3d052e6b67" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40215 } }
I20260620 20:21:08.251364 5693 raft_consensus.cc:388] T 00000000000000000000000000000000 P 670fdb5994a74b19a3517a3d052e6b67 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 20:21:08.251466 5693 raft_consensus.cc:482] T 00000000000000000000000000000000 P 670fdb5994a74b19a3517a3d052e6b67 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 20:21:08.251596 5693 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 670fdb5994a74b19a3517a3d052e6b67 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:21:08.253791 5693 raft_consensus.cc:504] T 00000000000000000000000000000000 P 670fdb5994a74b19a3517a3d052e6b67 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "670fdb5994a74b19a3517a3d052e6b67" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40215 } }
I20260620 20:21:08.254346 5693 leader_election.cc:302] T 00000000000000000000000000000000 P 670fdb5994a74b19a3517a3d052e6b67 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 670fdb5994a74b19a3517a3d052e6b67; no voters:
I20260620 20:21:08.254982 5693 leader_election.cc:288] T 00000000000000000000000000000000 P 670fdb5994a74b19a3517a3d052e6b67 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 20:21:08.255131 5703 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 670fdb5994a74b19a3517a3d052e6b67 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:21:08.255849 5703 raft_consensus.cc:686] T 00000000000000000000000000000000 P 670fdb5994a74b19a3517a3d052e6b67 [term 1 LEADER]: Becoming Leader. State: Replica: 670fdb5994a74b19a3517a3d052e6b67, State: Running, Role: LEADER
I20260620 20:21:08.256371 5703 consensus_queue.cc:237] T 00000000000000000000000000000000 P 670fdb5994a74b19a3517a3d052e6b67 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "670fdb5994a74b19a3517a3d052e6b67" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40215 } }
I20260620 20:21:08.257097 5693 sys_catalog.cc:564] T 00000000000000000000000000000000 P 670fdb5994a74b19a3517a3d052e6b67 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:21:08.259255 5705 sys_catalog.cc:455] T 00000000000000000000000000000000 P 670fdb5994a74b19a3517a3d052e6b67 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 670fdb5994a74b19a3517a3d052e6b67. Latest consensus state: current_term: 1 leader_uuid: "670fdb5994a74b19a3517a3d052e6b67" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "670fdb5994a74b19a3517a3d052e6b67" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40215 } } }
I20260620 20:21:08.259800 5705 sys_catalog.cc:458] T 00000000000000000000000000000000 P 670fdb5994a74b19a3517a3d052e6b67 [sys.catalog]: This master's current role is: LEADER
I20260620 20:21:08.259300 5704 sys_catalog.cc:455] T 00000000000000000000000000000000 P 670fdb5994a74b19a3517a3d052e6b67 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "670fdb5994a74b19a3517a3d052e6b67" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "670fdb5994a74b19a3517a3d052e6b67" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40215 } } }
I20260620 20:21:08.260784 5704 sys_catalog.cc:458] T 00000000000000000000000000000000 P 670fdb5994a74b19a3517a3d052e6b67 [sys.catalog]: This master's current role is: LEADER
I20260620 20:21:08.260939 5710 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 20:21:08.267397 5710 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 20:21:08.276757 5710 catalog_manager.cc:1347] Generated new cluster ID: b97b7771174c41e0958b3ba61e8a260f
I20260620 20:21:08.276960 5710 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 20:21:08.288862 5710 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 20:21:08.290292 5710 catalog_manager.cc:1504] Loading token signing keys...
I20260620 20:21:08.297732 5710 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 670fdb5994a74b19a3517a3d052e6b67: Generated new TSK 0
I20260620 20:21:08.298650 5710 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 20:21:08.306314 17252 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu pid 5626
W20260620 20:21:08.307660 5688 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:48737
W20260620 20:21:08.308470 17252 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu pid 5626: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 20:21:08.308627 17252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu with pid 5626
2026-06-20T20:21:08Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1873 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781986536687152-17252-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 20:21:08 dist-test-slave-2hpg krb5kdc[5724](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 20:21:08 dist-test-slave-2hpg krb5kdc[5724](info): set up 2 sockets
Jun 20 20:21:08 dist-test-slave-2hpg krb5kdc[5724](info): commencing operation
krb5kdc: starting...
W20260620 20:21:09.406493 17252 mini_kdc.cc:121] Time spent starting KDC: real 1.060s user 0.007s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781986536687152-17252-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781986536687152-17252-0/krb5kdc/test-user.keytab.
Jun 20 20:21:09 dist-test-slave-2hpg krb5kdc[5724](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986869, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T20:21:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T20:21:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781986536687152-17252-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781986536687152-17252-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 20:21:09.659399 17252 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
/tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781986536687152-17252-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781986536687152-17252-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781986536687152-17252-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781986536687152-17252-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39367
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781986536687152-17252-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44825
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39367
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781986536687152-17252-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781986536687152-17252-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781986536687152-17252-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:21:09.988766 5737 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:21:09.989187 5737 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:21:09.989363 5737 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:21:09.997143 5737 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:21:09.997282 5737 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:21:09.997365 5737 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:21:09.997448 5737 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:21:09.997512 5737 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:21:10.007658 5737 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44825
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781986536687152-17252-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781986536687152-17252-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39367
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781986536687152-17252-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39367
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781986536687152-17252-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781986536687152-17252-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:21:10.009040 5737 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:21:10.010492 5737 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:21:10.022771 5745 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:21:10.022866 5742 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:21:10.022817 5743 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:21:10.023584 5737 server_base.cc:1034] running on GCE node
I20260620 20:21:10.024350 5737 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:21:10.025748 5737 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:21:10.026954 5737 hybrid_clock.cc:648] HybridClock initialized: now 1781986870026906 us; error 36 us; skew 500 ppm
I20260620 20:21:10.027302 5737 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:21:10 dist-test-slave-2hpg krb5kdc[5724](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986870, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:21:10.030428 5737 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 20:21:10.031934 5737 webserver.cc:460] Webserver started at http://127.0.0.1:41479/ using document root <none> and password file <none>
I20260620 20:21:10.032609 5737 fs_manager.cc:359] Metadata directory not provided
I20260620 20:21:10.032753 5737 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:21:10.033066 5737 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:21:10.036163 5737 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781986536687152-17252-0/minicluster-data/master-0/data/instance:
uuid: "adbec7ec93a341cdb44a444dfa528170"
format_stamp: "Formatted at 2026-06-20 20:21:10 on dist-test-slave-2hpg"
server_key: "26b51fad97c42a8eaa0647c590eef77a"
server_key_iv: "212facbb2325b9f86ea2d8cd355026d3"
server_key_version: "encryptionkey@0"
I20260620 20:21:10.036935 5737 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781986536687152-17252-0/minicluster-data/master-0/wal/instance:
uuid: "adbec7ec93a341cdb44a444dfa528170"
format_stamp: "Formatted at 2026-06-20 20:21:10 on dist-test-slave-2hpg"
server_key: "26b51fad97c42a8eaa0647c590eef77a"
server_key_iv: "212facbb2325b9f86ea2d8cd355026d3"
server_key_version: "encryptionkey@0"
I20260620 20:21:10.041981 5737 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260620 20:21:10.045405 5752 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:21:10.046839 5737 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260620 20:21:10.047039 5737 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781986536687152-17252-0/minicluster-data/master-0/data,/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781986536687152-17252-0/minicluster-data/master-0/wal
uuid: "adbec7ec93a341cdb44a444dfa528170"
format_stamp: "Formatted at 2026-06-20 20:21:10 on dist-test-slave-2hpg"
server_key: "26b51fad97c42a8eaa0647c590eef77a"
server_key_iv: "212facbb2325b9f86ea2d8cd355026d3"
server_key_version: "encryptionkey@0"
I20260620 20:21:10.047232 5737 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781986536687152-17252-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781986536687152-17252-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781986536687152-17252-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:21:10.053023 5737 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:21:10.056857 5737 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:21:10.057145 5737 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:21:10.075434 5737 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39367
I20260620 20:21:10.075518 5803 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39367 every 8 connection(s)
I20260620 20:21:10.076779 5737 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781986536687152-17252-0/minicluster-data/master-0/data/info.pb
I20260620 20:21:10.080353 5804 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:21:10.082479 17252 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu as pid 5737
I20260620 20:21:10.082683 17252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781986536687152-17252-0/minicluster-data/master-0/wal/instance
I20260620 20:21:10.083112 17252 external_mini_cluster.cc:1428] Setting key 0c9f3587bdee00a4802c6defbac4dd50
I20260620 20:21:10.090736 5807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:10.084810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50449 (local address 127.0.0.1:39367)
0620 20:21:10.085557 (+ 747us) server_negotiation.cc:206] Beginning negotiation
0620 20:21:10.085569 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 20:21:10.085615 (+ 46us) server_negotiation.cc:407] Connection header received
0620 20:21:10.087285 (+ 1670us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:21:10.087306 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:21:10.087624 (+ 318us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:21:10.087991 (+ 367us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:21:10.089057 (+ 1066us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:21:10.089063 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:21:10.089075 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:21:10.089091 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:21:10.089148 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:21:10.089170 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:21:10.089437 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0620 20:21:10.089609 (+ 172us) server_negotiation.cc:299] Negotiation successful
0620 20:21:10.089786 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":540,"thread_start_us":321,"threads_started":1}
I20260620 20:21:10.091924 5804 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P adbec7ec93a341cdb44a444dfa528170: Bootstrap starting.
I20260620 20:21:10.094246 17252 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 20:21:10.095147 5804 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P adbec7ec93a341cdb44a444dfa528170: Neither blocks nor log segments found. Creating new log.
I20260620 20:21:10.096122 5804 log.cc:824] T 00000000000000000000000000000000 P adbec7ec93a341cdb44a444dfa528170: Log is configured to *not* fsync() on all Append() calls
I20260620 20:21:10.099095 5804 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P adbec7ec93a341cdb44a444dfa528170: No bootstrap required, opened a new log
I20260620 20:21:10.101054 5807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:10.098602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48613 (local address 127.0.0.1:39367)
0620 20:21:10.098796 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0620 20:21:10.098802 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:21:10.099136 (+ 334us) server_negotiation.cc:407] Connection header received
0620 20:21:10.099322 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:21:10.099325 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:21:10.099368 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:21:10.099461 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:21:10.100384 (+ 923us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:21:10.100389 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:21:10.100392 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:21:10.100406 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:21:10.100445 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:21:10.100459 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:21:10.100544 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0620 20:21:10.100693 (+ 149us) server_negotiation.cc:299] Negotiation successful
0620 20:21:10.100736 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 20:21:10.107436 5804 raft_consensus.cc:348] T 00000000000000000000000000000000 P adbec7ec93a341cdb44a444dfa528170 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "adbec7ec93a341cdb44a444dfa528170" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39367 } }
I20260620 20:21:10.107790 5804 raft_consensus.cc:374] T 00000000000000000000000000000000 P adbec7ec93a341cdb44a444dfa528170 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:21:10.107905 5804 raft_consensus.cc:729] T 00000000000000000000000000000000 P adbec7ec93a341cdb44a444dfa528170 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: adbec7ec93a341cdb44a444dfa528170, State: Initialized, Role: FOLLOWER
I20260620 20:21:10.108438 5804 consensus_queue.cc:260] T 00000000000000000000000000000000 P adbec7ec93a341cdb44a444dfa528170 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "adbec7ec93a341cdb44a444dfa528170" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39367 } }
I20260620 20:21:10.108676 5804 raft_consensus.cc:388] T 00000000000000000000000000000000 P adbec7ec93a341cdb44a444dfa528170 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 20:21:10.108826 5804 raft_consensus.cc:482] T 00000000000000000000000000000000 P adbec7ec93a341cdb44a444dfa528170 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 20:21:10.109001 5804 raft_consensus.cc:3037] T 00000000000000000000000000000000 P adbec7ec93a341cdb44a444dfa528170 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:21:10.111403 5804 raft_consensus.cc:504] T 00000000000000000000000000000000 P adbec7ec93a341cdb44a444dfa528170 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "adbec7ec93a341cdb44a444dfa528170" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39367 } }
I20260620 20:21:10.111898 5804 leader_election.cc:302] T 00000000000000000000000000000000 P adbec7ec93a341cdb44a444dfa528170 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: adbec7ec93a341cdb44a444dfa528170; no voters:
I20260620 20:21:10.112432 5804 leader_election.cc:288] T 00000000000000000000000000000000 P adbec7ec93a341cdb44a444dfa528170 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 20:21:10.112576 5814 raft_consensus.cc:2781] T 00000000000000000000000000000000 P adbec7ec93a341cdb44a444dfa528170 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:21:10.113377 5814 raft_consensus.cc:686] T 00000000000000000000000000000000 P adbec7ec93a341cdb44a444dfa528170 [term 1 LEADER]: Becoming Leader. State: Replica: adbec7ec93a341cdb44a444dfa528170, State: Running, Role: LEADER
I20260620 20:21:10.114055 5814 consensus_queue.cc:237] T 00000000000000000000000000000000 P adbec7ec93a341cdb44a444dfa528170 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "adbec7ec93a341cdb44a444dfa528170" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39367 } }
I20260620 20:21:10.114662 5804 sys_catalog.cc:564] T 00000000000000000000000000000000 P adbec7ec93a341cdb44a444dfa528170 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:21:10.117159 5815 sys_catalog.cc:455] T 00000000000000000000000000000000 P adbec7ec93a341cdb44a444dfa528170 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "adbec7ec93a341cdb44a444dfa528170" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "adbec7ec93a341cdb44a444dfa528170" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39367 } } }
I20260620 20:21:10.117129 5816 sys_catalog.cc:455] T 00000000000000000000000000000000 P adbec7ec93a341cdb44a444dfa528170 [sys.catalog]: SysCatalogTable state changed. Reason: New leader adbec7ec93a341cdb44a444dfa528170. Latest consensus state: current_term: 1 leader_uuid: "adbec7ec93a341cdb44a444dfa528170" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "adbec7ec93a341cdb44a444dfa528170" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39367 } } }
I20260620 20:21:10.117745 5815 sys_catalog.cc:458] T 00000000000000000000000000000000 P adbec7ec93a341cdb44a444dfa528170 [sys.catalog]: This master's current role is: LEADER
I20260620 20:21:10.117825 5816 sys_catalog.cc:458] T 00000000000000000000000000000000 P adbec7ec93a341cdb44a444dfa528170 [sys.catalog]: This master's current role is: LEADER
I20260620 20:21:10.119247 5820 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 20:21:10.124217 5820 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 20:21:10.132516 5820 catalog_manager.cc:1347] Generated new cluster ID: 17eb983b92d349b2affb8c63fdedac8d
I20260620 20:21:10.132632 5820 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 20:21:10.156558 5820 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 20:21:10.157069 5820 catalog_manager.cc:1504] Loading token signing keys...
I20260620 20:21:10.175135 5820 catalog_manager.cc:5868] T 00000000000000000000000000000000 P adbec7ec93a341cdb44a444dfa528170: Generated new TSK 0
I20260620 20:21:10.176187 5820 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 20:21:10.192582 17252 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu pid 5737
W20260620 20:21:10.193698 5799 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:50449
W20260620 20:21:10.194442 17252 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu pid 5737: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 20:21:10.194599 17252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu with pid 5737
2026-06-20T20:21:10Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1883 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781986536687152-17252-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 20:21:10 dist-test-slave-2hpg krb5kdc[5835](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 20:21:10 dist-test-slave-2hpg krb5kdc[5835](info): set up 2 sockets
Jun 20 20:21:10 dist-test-slave-2hpg krb5kdc[5835](info): commencing operation
krb5kdc: starting...
W20260620 20:21:11.298606 17252 mini_kdc.cc:121] Time spent starting KDC: real 1.068s user 0.008s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781986536687152-17252-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781986536687152-17252-0/krb5kdc/test-user.keytab.
Jun 20 20:21:11 dist-test-slave-2hpg krb5kdc[5835](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986871, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T20:21:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T20:21:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781986536687152-17252-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781986536687152-17252-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 20:21:11.556944 17252 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
/tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781986536687152-17252-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781986536687152-17252-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781986536687152-17252-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781986536687152-17252-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35853
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781986536687152-17252-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33299
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35853
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781986536687152-17252-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781986536687152-17252-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781986536687152-17252-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:21:11.888082 5848 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:21:11.888422 5848 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:21:11.888598 5848 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:21:11.896395 5848 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:21:11.896504 5848 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:21:11.896586 5848 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:21:11.896697 5848 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:21:11.896763 5848 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:21:11.906517 5848 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33299
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781986536687152-17252-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781986536687152-17252-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35853
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781986536687152-17252-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35853
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781986536687152-17252-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781986536687152-17252-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:21:11.907758 5848 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:21:11.909041 5848 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:21:11.921588 5853 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:21:11.921502 5854 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:21:11.921674 5856 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:21:11.922444 5848 server_base.cc:1034] running on GCE node
I20260620 20:21:11.923357 5848 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:21:11.924783 5848 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:21:11.925997 5848 hybrid_clock.cc:648] HybridClock initialized: now 1781986871925942 us; error 45 us; skew 500 ppm
I20260620 20:21:11.926311 5848 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:21:11 dist-test-slave-2hpg krb5kdc[5835](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986871, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:21:11.929396 5848 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 20:21:11.930892 5848 webserver.cc:460] Webserver started at http://127.0.0.1:42995/ using document root <none> and password file <none>
I20260620 20:21:11.931551 5848 fs_manager.cc:359] Metadata directory not provided
I20260620 20:21:11.931682 5848 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:21:11.931979 5848 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:21:11.934926 5848 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781986536687152-17252-0/minicluster-data/master-0/data/instance:
uuid: "273a026fe0c84088acd66f7f656060e0"
format_stamp: "Formatted at 2026-06-20 20:21:11 on dist-test-slave-2hpg"
server_key: "2ede06a404d22e1738e9ad000af3bfd6"
server_key_iv: "a3f9dd618dd086b073375bea39d6b707"
server_key_version: "encryptionkey@0"
I20260620 20:21:11.935693 5848 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781986536687152-17252-0/minicluster-data/master-0/wal/instance:
uuid: "273a026fe0c84088acd66f7f656060e0"
format_stamp: "Formatted at 2026-06-20 20:21:11 on dist-test-slave-2hpg"
server_key: "2ede06a404d22e1738e9ad000af3bfd6"
server_key_iv: "a3f9dd618dd086b073375bea39d6b707"
server_key_version: "encryptionkey@0"
I20260620 20:21:11.940709 5848 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260620 20:21:11.944049 5863 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:21:11.945268 5848 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260620 20:21:11.945453 5848 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781986536687152-17252-0/minicluster-data/master-0/data,/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781986536687152-17252-0/minicluster-data/master-0/wal
uuid: "273a026fe0c84088acd66f7f656060e0"
format_stamp: "Formatted at 2026-06-20 20:21:11 on dist-test-slave-2hpg"
server_key: "2ede06a404d22e1738e9ad000af3bfd6"
server_key_iv: "a3f9dd618dd086b073375bea39d6b707"
server_key_version: "encryptionkey@0"
I20260620 20:21:11.945643 5848 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781986536687152-17252-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781986536687152-17252-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781986536687152-17252-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:21:11.962376 5848 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:21:11.966338 5848 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:21:11.966621 5848 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:21:11.985283 5848 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35853
I20260620 20:21:11.985307 5914 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35853 every 8 connection(s)
I20260620 20:21:11.986676 5848 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781986536687152-17252-0/minicluster-data/master-0/data/info.pb
I20260620 20:21:11.990394 5915 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:21:11.991456 17252 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu as pid 5848
I20260620 20:21:11.991645 17252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781986536687152-17252-0/minicluster-data/master-0/wal/instance
I20260620 20:21:11.992076 17252 external_mini_cluster.cc:1428] Setting key 04f42c8e2ef8043d12c3872a20d995fc
I20260620 20:21:12.003718 5915 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 273a026fe0c84088acd66f7f656060e0: Bootstrap starting.
Jun 20 20:21:12 dist-test-slave-2hpg krb5kdc[5835](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986871, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 20:21:12.007212 5915 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 273a026fe0c84088acd66f7f656060e0: Neither blocks nor log segments found. Creating new log.
I20260620 20:21:12.008381 5915 log.cc:824] T 00000000000000000000000000000000 P 273a026fe0c84088acd66f7f656060e0: Log is configured to *not* fsync() on all Append() calls
I20260620 20:21:12.011582 5915 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 273a026fe0c84088acd66f7f656060e0: No bootstrap required, opened a new log
I20260620 20:21:12.015776 5918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:11.993765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.111:53297 (local address 127.0.0.1:35853)
0620 20:21:11.994603 (+ 838us) server_negotiation.cc:206] Beginning negotiation
0620 20:21:11.994614 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 20:21:11.994656 (+ 42us) server_negotiation.cc:407] Connection header received
0620 20:21:11.996659 (+ 2003us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:21:11.996693 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:21:11.997245 (+ 552us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:21:11.997741 (+ 496us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:21:11.999614 (+ 1873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:12.000910 (+ 1296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:12.002251 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:12.002942 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:12.003147 (+ 205us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:21:12.007663 (+ 4516us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:21:12.007673 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:21:12.007690 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:21:12.007729 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:21:12.011206 (+ 3477us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:21:12.012187 (+ 981us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:21:12.012193 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:21:12.012200 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:21:12.012365 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:21:12.012893 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:21:12.012896 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:21:12.012899 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:21:12.013439 (+ 540us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:21:12.013704 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0620 20:21:12.014297 (+ 593us) server_negotiation.cc:299] Negotiation successful
0620 20:21:12.014720 (+ 423us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":625,"thread_start_us":355,"threads_started":1}
I20260620 20:21:12.019860 17252 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 20:21:12.020453 5915 raft_consensus.cc:348] T 00000000000000000000000000000000 P 273a026fe0c84088acd66f7f656060e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "273a026fe0c84088acd66f7f656060e0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35853 } }
I20260620 20:21:12.020866 5915 raft_consensus.cc:374] T 00000000000000000000000000000000 P 273a026fe0c84088acd66f7f656060e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:21:12.020964 5915 raft_consensus.cc:729] T 00000000000000000000000000000000 P 273a026fe0c84088acd66f7f656060e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 273a026fe0c84088acd66f7f656060e0, State: Initialized, Role: FOLLOWER
I20260620 20:21:12.021597 5915 consensus_queue.cc:260] T 00000000000000000000000000000000 P 273a026fe0c84088acd66f7f656060e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "273a026fe0c84088acd66f7f656060e0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35853 } }
I20260620 20:21:12.021907 5915 raft_consensus.cc:388] T 00000000000000000000000000000000 P 273a026fe0c84088acd66f7f656060e0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 20:21:12.022048 5915 raft_consensus.cc:482] T 00000000000000000000000000000000 P 273a026fe0c84088acd66f7f656060e0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 20:21:12.022215 5915 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 273a026fe0c84088acd66f7f656060e0 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:21:12.025431 5915 raft_consensus.cc:504] T 00000000000000000000000000000000 P 273a026fe0c84088acd66f7f656060e0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "273a026fe0c84088acd66f7f656060e0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35853 } }
I20260620 20:21:12.026006 5915 leader_election.cc:302] T 00000000000000000000000000000000 P 273a026fe0c84088acd66f7f656060e0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 273a026fe0c84088acd66f7f656060e0; no voters:
I20260620 20:21:12.026607 5915 leader_election.cc:288] T 00000000000000000000000000000000 P 273a026fe0c84088acd66f7f656060e0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 20:21:12.026830 5925 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 273a026fe0c84088acd66f7f656060e0 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:21:12.027556 5925 raft_consensus.cc:686] T 00000000000000000000000000000000 P 273a026fe0c84088acd66f7f656060e0 [term 1 LEADER]: Becoming Leader. State: Replica: 273a026fe0c84088acd66f7f656060e0, State: Running, Role: LEADER
I20260620 20:21:12.028234 5925 consensus_queue.cc:237] T 00000000000000000000000000000000 P 273a026fe0c84088acd66f7f656060e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "273a026fe0c84088acd66f7f656060e0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35853 } }
I20260620 20:21:12.030019 5925 sys_catalog.cc:455] T 00000000000000000000000000000000 P 273a026fe0c84088acd66f7f656060e0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 273a026fe0c84088acd66f7f656060e0. Latest consensus state: current_term: 1 leader_uuid: "273a026fe0c84088acd66f7f656060e0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "273a026fe0c84088acd66f7f656060e0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35853 } } }
I20260620 20:21:12.030378 5925 sys_catalog.cc:458] T 00000000000000000000000000000000 P 273a026fe0c84088acd66f7f656060e0 [sys.catalog]: This master's current role is: LEADER
I20260620 20:21:12.030861 5927 sys_catalog.cc:455] T 00000000000000000000000000000000 P 273a026fe0c84088acd66f7f656060e0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "273a026fe0c84088acd66f7f656060e0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "273a026fe0c84088acd66f7f656060e0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35853 } } }
I20260620 20:21:12.031234 5927 sys_catalog.cc:458] T 00000000000000000000000000000000 P 273a026fe0c84088acd66f7f656060e0 [sys.catalog]: This master's current role is: LEADER
I20260620 20:21:12.034924 5915 sys_catalog.cc:564] T 00000000000000000000000000000000 P 273a026fe0c84088acd66f7f656060e0 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:21:12.037488 5929 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 20:21:12.043498 5929 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 20:21:12.047367 5918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:12.025122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.111:49483 (local address 127.0.0.1:35853)
0620 20:21:12.025326 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0620 20:21:12.025332 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:21:12.027345 (+ 2013us) server_negotiation.cc:407] Connection header received
0620 20:21:12.027676 (+ 331us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:21:12.027684 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:21:12.027790 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:21:12.027874 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:21:12.030488 (+ 2614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:12.032134 (+ 1646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:12.034178 (+ 2044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:12.034892 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:12.035016 (+ 124us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:21:12.038918 (+ 3902us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:21:12.038926 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:21:12.038932 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:21:12.038968 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:21:12.042942 (+ 3974us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:21:12.044017 (+ 1075us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:21:12.044022 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:21:12.044025 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:21:12.044161 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:21:12.044805 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:21:12.044809 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:21:12.044812 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:21:12.045243 (+ 431us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:21:12.046590 (+ 1347us) server_negotiation.cc:1035] Waiting for connection context
0620 20:21:12.046783 (+ 193us) server_negotiation.cc:299] Negotiation successful
0620 20:21:12.046970 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 20:21:12.059870 5929 catalog_manager.cc:1347] Generated new cluster ID: 521fb1bd864a4e36b287ae1d49231e63
I20260620 20:21:12.060029 5929 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 20:21:12.074177 5929 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 20:21:12.075459 5929 catalog_manager.cc:1504] Loading token signing keys...
I20260620 20:21:12.084757 5929 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 273a026fe0c84088acd66f7f656060e0: Generated new TSK 0
I20260620 20:21:12.085826 5929 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 20:21:12.098685 17252 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu pid 5848
I20260620 20:21:12.100281 5910 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:21:12.192063 17252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu with pid 5848
W20260620 20:21:12.208837 5833 connection.cc:535] client connection to 127.0.0.1:35853 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35853): Connection reset by peer (error 104)
2026-06-20T20:21:12Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1997 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781986536687152-17252-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 20:21:12 dist-test-slave-2hpg krb5kdc[5947](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 20:21:12 dist-test-slave-2hpg krb5kdc[5947](info): set up 2 sockets
Jun 20 20:21:12 dist-test-slave-2hpg krb5kdc[5947](info): commencing operation
krb5kdc: starting...
W20260620 20:21:13.318718 17252 mini_kdc.cc:121] Time spent starting KDC: real 1.091s user 0.006s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781986536687152-17252-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781986536687152-17252-0/krb5kdc/test-user.keytab.
Jun 20 20:21:13 dist-test-slave-2hpg krb5kdc[5947](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986873, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T20:21:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T20:21:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781986536687152-17252-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781986536687152-17252-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 20:21:13.570214 17252 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
/tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781986536687152-17252-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781986536687152-17252-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781986536687152-17252-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781986536687152-17252-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41241
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781986536687152-17252-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42469
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41241
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781986536687152-17252-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781986536687152-17252-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781986536687152-17252-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:21:13.897205 5960 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:21:13.897500 5960 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:21:13.897706 5960 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:21:13.904992 5960 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:21:13.905098 5960 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:21:13.905205 5960 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:21:13.905313 5960 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:21:13.905373 5960 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:21:13.915364 5960 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42469
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781986536687152-17252-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781986536687152-17252-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41241
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781986536687152-17252-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41241
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781986536687152-17252-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781986536687152-17252-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:21:13.916695 5960 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:21:13.918081 5960 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:21:13.929929 5966 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:21:13.930064 5965 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:21:13.930028 5968 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:21:13.931016 5960 server_base.cc:1034] running on GCE node
I20260620 20:21:13.931916 5960 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:21:13.933439 5960 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:21:13.934682 5960 hybrid_clock.cc:648] HybridClock initialized: now 1781986873934649 us; error 61 us; skew 500 ppm
I20260620 20:21:13.935065 5960 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:21:13 dist-test-slave-2hpg krb5kdc[5947](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986873, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:21:13.942986 5960 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 20:21:13.944520 5960 webserver.cc:460] Webserver started at http://127.0.0.1:43139/ using document root <none> and password file <none>
I20260620 20:21:13.945215 5960 fs_manager.cc:359] Metadata directory not provided
I20260620 20:21:13.945366 5960 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:21:13.945690 5960 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:21:13.948735 5960 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781986536687152-17252-0/minicluster-data/master-0/data/instance:
uuid: "931502be26f242259384f508a9194a11"
format_stamp: "Formatted at 2026-06-20 20:21:13 on dist-test-slave-2hpg"
server_key: "c1e55bab8e8c00acaa22a9f004f4f10e"
server_key_iv: "ec79f3f882fa6e1d23f001930155e082"
server_key_version: "encryptionkey@0"
I20260620 20:21:13.949465 5960 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781986536687152-17252-0/minicluster-data/master-0/wal/instance:
uuid: "931502be26f242259384f508a9194a11"
format_stamp: "Formatted at 2026-06-20 20:21:13 on dist-test-slave-2hpg"
server_key: "c1e55bab8e8c00acaa22a9f004f4f10e"
server_key_iv: "ec79f3f882fa6e1d23f001930155e082"
server_key_version: "encryptionkey@0"
I20260620 20:21:13.954849 5960 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260620 20:21:13.958302 5975 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:21:13.959606 5960 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 20:21:13.959789 5960 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781986536687152-17252-0/minicluster-data/master-0/data,/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781986536687152-17252-0/minicluster-data/master-0/wal
uuid: "931502be26f242259384f508a9194a11"
format_stamp: "Formatted at 2026-06-20 20:21:13 on dist-test-slave-2hpg"
server_key: "c1e55bab8e8c00acaa22a9f004f4f10e"
server_key_iv: "ec79f3f882fa6e1d23f001930155e082"
server_key_version: "encryptionkey@0"
I20260620 20:21:13.959986 5960 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781986536687152-17252-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781986536687152-17252-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781986536687152-17252-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:21:13.974601 5960 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:21:13.978788 5960 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:21:13.979097 5960 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:21:13.997991 5960 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41241
I20260620 20:21:13.997985 6026 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41241 every 8 connection(s)
I20260620 20:21:13.999435 5960 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781986536687152-17252-0/minicluster-data/master-0/data/info.pb
I20260620 20:21:14.003116 17252 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu as pid 5960
I20260620 20:21:14.003233 6027 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:21:14.003319 17252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781986536687152-17252-0/minicluster-data/master-0/wal/instance
I20260620 20:21:14.003852 17252 external_mini_cluster.cc:1428] Setting key ebcf7181a4a62a86800883da2ededb24
I20260620 20:21:14.018695 6030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:14.005828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.111:36619 (local address 127.0.0.1:41241)
0620 20:21:14.006936 (+ 1108us) server_negotiation.cc:206] Beginning negotiation
0620 20:21:14.006952 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 20:21:14.007003 (+ 51us) server_negotiation.cc:407] Connection header received
0620 20:21:14.009184 (+ 2181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:21:14.009208 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:21:14.009651 (+ 443us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:21:14.010093 (+ 442us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:21:14.010655 (+ 562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:14.012026 (+ 1371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:14.013922 (+ 1896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:14.014697 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:14.014826 (+ 129us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:21:14.016237 (+ 1411us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:21:14.016244 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:21:14.016258 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:21:14.016840 (+ 582us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:21:14.016924 (+ 84us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:21:14.016949 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:21:14.017080 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0620 20:21:14.017402 (+ 322us) server_negotiation.cc:299] Negotiation successful
0620 20:21:14.017645 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":741,"thread_start_us":413,"threads_started":1}
I20260620 20:21:14.018792 6027 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 931502be26f242259384f508a9194a11: Bootstrap starting.
I20260620 20:21:14.022075 6027 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 931502be26f242259384f508a9194a11: Neither blocks nor log segments found. Creating new log.
I20260620 20:21:14.022645 17252 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 20:21:14.023283 6027 log.cc:824] T 00000000000000000000000000000000 P 931502be26f242259384f508a9194a11: Log is configured to *not* fsync() on all Append() calls
I20260620 20:21:14.026580 6027 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 931502be26f242259384f508a9194a11: No bootstrap required, opened a new log
I20260620 20:21:14.034789 6030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:14.027891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.111:55887 (local address 127.0.0.1:41241)
0620 20:21:14.028068 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0620 20:21:14.028074 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:21:14.028526 (+ 452us) server_negotiation.cc:407] Connection header received
0620 20:21:14.028683 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:21:14.028688 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:21:14.028766 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:21:14.028869 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:21:14.029386 (+ 517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:14.030600 (+ 1214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:14.031937 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:14.032665 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:14.032798 (+ 133us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:21:14.033975 (+ 1177us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:21:14.033983 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:21:14.033992 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:21:14.034020 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:21:14.034068 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:21:14.034093 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:21:14.034214 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0620 20:21:14.034379 (+ 165us) server_negotiation.cc:299] Negotiation successful
0620 20:21:14.034432 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 20:21:14.036542 6027 raft_consensus.cc:348] T 00000000000000000000000000000000 P 931502be26f242259384f508a9194a11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "931502be26f242259384f508a9194a11" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41241 } }
I20260620 20:21:14.036899 6027 raft_consensus.cc:374] T 00000000000000000000000000000000 P 931502be26f242259384f508a9194a11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:21:14.037009 6027 raft_consensus.cc:729] T 00000000000000000000000000000000 P 931502be26f242259384f508a9194a11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 931502be26f242259384f508a9194a11, State: Initialized, Role: FOLLOWER
I20260620 20:21:14.037585 6027 consensus_queue.cc:260] T 00000000000000000000000000000000 P 931502be26f242259384f508a9194a11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "931502be26f242259384f508a9194a11" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41241 } }
I20260620 20:21:14.037819 6027 raft_consensus.cc:388] T 00000000000000000000000000000000 P 931502be26f242259384f508a9194a11 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 20:21:14.038019 6027 raft_consensus.cc:482] T 00000000000000000000000000000000 P 931502be26f242259384f508a9194a11 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 20:21:14.038184 6027 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 931502be26f242259384f508a9194a11 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:21:14.040769 6027 raft_consensus.cc:504] T 00000000000000000000000000000000 P 931502be26f242259384f508a9194a11 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "931502be26f242259384f508a9194a11" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41241 } }
I20260620 20:21:14.041283 6027 leader_election.cc:302] T 00000000000000000000000000000000 P 931502be26f242259384f508a9194a11 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 931502be26f242259384f508a9194a11; no voters:
I20260620 20:21:14.041851 6027 leader_election.cc:288] T 00000000000000000000000000000000 P 931502be26f242259384f508a9194a11 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 20:21:14.042075 6037 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 931502be26f242259384f508a9194a11 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:21:14.042865 6037 raft_consensus.cc:686] T 00000000000000000000000000000000 P 931502be26f242259384f508a9194a11 [term 1 LEADER]: Becoming Leader. State: Replica: 931502be26f242259384f508a9194a11, State: Running, Role: LEADER
I20260620 20:21:14.043424 6037 consensus_queue.cc:237] T 00000000000000000000000000000000 P 931502be26f242259384f508a9194a11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "931502be26f242259384f508a9194a11" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41241 } }
I20260620 20:21:14.044205 6027 sys_catalog.cc:564] T 00000000000000000000000000000000 P 931502be26f242259384f508a9194a11 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:21:14.047454 6039 sys_catalog.cc:455] T 00000000000000000000000000000000 P 931502be26f242259384f508a9194a11 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 931502be26f242259384f508a9194a11. Latest consensus state: current_term: 1 leader_uuid: "931502be26f242259384f508a9194a11" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "931502be26f242259384f508a9194a11" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41241 } } }
I20260620 20:21:14.047847 6039 sys_catalog.cc:458] T 00000000000000000000000000000000 P 931502be26f242259384f508a9194a11 [sys.catalog]: This master's current role is: LEADER
I20260620 20:21:14.047546 6038 sys_catalog.cc:455] T 00000000000000000000000000000000 P 931502be26f242259384f508a9194a11 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "931502be26f242259384f508a9194a11" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "931502be26f242259384f508a9194a11" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41241 } } }
I20260620 20:21:14.047955 6038 sys_catalog.cc:458] T 00000000000000000000000000000000 P 931502be26f242259384f508a9194a11 [sys.catalog]: This master's current role is: LEADER
I20260620 20:21:14.049539 6045 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 20:21:14.055145 6045 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 20:21:14.064332 6045 catalog_manager.cc:1347] Generated new cluster ID: ffd5eed3b93f4b92b732747afdabb48c
I20260620 20:21:14.064457 6045 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 20:21:14.071856 6045 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 20:21:14.072997 6045 catalog_manager.cc:1504] Loading token signing keys...
I20260620 20:21:14.083529 6045 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 931502be26f242259384f508a9194a11: Generated new TSK 0
I20260620 20:21:14.084297 6045 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 20:21:14.098570 17252 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu pid 5960
W20260620 20:21:14.099686 6022 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.111:36619
W20260620 20:21:14.100531 17252 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu pid 5960: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 20:21:14.100687 17252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu with pid 5960
W20260620 20:21:14.117602 5945 connection.cc:535] client connection to 127.0.0.1:41241 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41241): Connection reset by peer (error 104)
2026-06-20T20:21:14Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1908 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781986536687152-17252-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 20:21:14 dist-test-slave-2hpg krb5kdc[6058](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 20:21:14 dist-test-slave-2hpg krb5kdc[6058](info): set up 2 sockets
Jun 20 20:21:14 dist-test-slave-2hpg krb5kdc[6058](info): commencing operation
krb5kdc: starting...
W20260620 20:21:15.214617 17252 mini_kdc.cc:121] Time spent starting KDC: real 1.078s user 0.000s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781986536687152-17252-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781986536687152-17252-0/krb5kdc/test-user.keytab.
Jun 20 20:21:15 dist-test-slave-2hpg krb5kdc[6058](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986875, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T20:21:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T20:21:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781986536687152-17252-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781986536687152-17252-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 20:21:15.468748 17252 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
/tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781986536687152-17252-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781986536687152-17252-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781986536687152-17252-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781986536687152-17252-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43491
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781986536687152-17252-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43491
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781986536687152-17252-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781986536687152-17252-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781986536687152-17252-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:21:15.805352 6071 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:21:15.805646 6071 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:21:15.805826 6071 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:21:15.813113 6071 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:21:15.813225 6071 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:21:15.813357 6071 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:21:15.813439 6071 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:21:15.813505 6071 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:21:15.823530 6071 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781986536687152-17252-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781986536687152-17252-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43491
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781986536687152-17252-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43491
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781986536687152-17252-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781986536687152-17252-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:21:15.824823 6071 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:21:15.826172 6071 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:21:15.837831 6077 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:21:15.837915 6076 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:21:15.837939 6079 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:21:15.838876 6071 server_base.cc:1034] running on GCE node
I20260620 20:21:15.839588 6071 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:21:15.840925 6071 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:21:15.842204 6071 hybrid_clock.cc:648] HybridClock initialized: now 1781986875842147 us; error 40 us; skew 500 ppm
I20260620 20:21:15.842618 6071 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:21:15 dist-test-slave-2hpg krb5kdc[6058](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986875, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:21:15.845903 6071 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 20:21:15.847352 6071 webserver.cc:460] Webserver started at http://127.0.0.1:45919/ using document root <none> and password file <none>
I20260620 20:21:15.848016 6071 fs_manager.cc:359] Metadata directory not provided
I20260620 20:21:15.848116 6071 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:21:15.848358 6071 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:21:15.851455 6071 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781986536687152-17252-0/minicluster-data/master-0/data/instance:
uuid: "499f46a5e8d9413aa9b464df6b14e5f5"
format_stamp: "Formatted at 2026-06-20 20:21:15 on dist-test-slave-2hpg"
server_key: "86aa8016eb13f5ff8ab1d9140878458a"
server_key_iv: "4cba65c39f3c619e068a90bb15ace269"
server_key_version: "encryptionkey@0"
I20260620 20:21:15.852226 6071 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781986536687152-17252-0/minicluster-data/master-0/wal/instance:
uuid: "499f46a5e8d9413aa9b464df6b14e5f5"
format_stamp: "Formatted at 2026-06-20 20:21:15 on dist-test-slave-2hpg"
server_key: "86aa8016eb13f5ff8ab1d9140878458a"
server_key_iv: "4cba65c39f3c619e068a90bb15ace269"
server_key_version: "encryptionkey@0"
I20260620 20:21:15.857671 6071 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260620 20:21:15.861228 6086 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:21:15.862701 6071 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260620 20:21:15.862846 6071 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781986536687152-17252-0/minicluster-data/master-0/data,/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781986536687152-17252-0/minicluster-data/master-0/wal
uuid: "499f46a5e8d9413aa9b464df6b14e5f5"
format_stamp: "Formatted at 2026-06-20 20:21:15 on dist-test-slave-2hpg"
server_key: "86aa8016eb13f5ff8ab1d9140878458a"
server_key_iv: "4cba65c39f3c619e068a90bb15ace269"
server_key_version: "encryptionkey@0"
I20260620 20:21:15.863029 6071 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781986536687152-17252-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781986536687152-17252-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781986536687152-17252-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:21:15.868727 6071 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:21:15.872799 6071 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:21:15.873062 6071 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:21:15.891330 6071 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43491
I20260620 20:21:15.891371 6137 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43491 every 8 connection(s)
I20260620 20:21:15.892827 6071 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781986536687152-17252-0/minicluster-data/master-0/data/info.pb
I20260620 20:21:15.894230 17252 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu as pid 6071
I20260620 20:21:15.894404 17252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781986536687152-17252-0/minicluster-data/master-0/wal/instance
I20260620 20:21:15.894830 17252 external_mini_cluster.cc:1428] Setting key ac80aa3cc139dfd5a09bf33e22526fa0
I20260620 20:21:15.897099 6138 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:21:15.902639 6141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:15.896555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.111:38669 (local address 127.0.0.1:43491)
0620 20:21:15.897352 (+ 797us) server_negotiation.cc:206] Beginning negotiation
0620 20:21:15.897364 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 20:21:15.897409 (+ 45us) server_negotiation.cc:407] Connection header received
0620 20:21:15.899140 (+ 1731us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:21:15.899162 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:21:15.899499 (+ 337us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:21:15.899913 (+ 414us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:21:15.900957 (+ 1044us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:21:15.900963 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:21:15.900974 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:21:15.900987 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:21:15.901075 (+ 88us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:21:15.901097 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:21:15.901395 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0620 20:21:15.901536 (+ 141us) server_negotiation.cc:299] Negotiation successful
0620 20:21:15.901714 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":530,"thread_start_us":304,"threads_started":1}
I20260620 20:21:15.906629 17252 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 20:21:15.908798 6138 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 499f46a5e8d9413aa9b464df6b14e5f5: Bootstrap starting.
I20260620 20:21:15.912398 6138 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 499f46a5e8d9413aa9b464df6b14e5f5: Neither blocks nor log segments found. Creating new log.
I20260620 20:21:15.913587 6141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:15.911082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.111:37213 (local address 127.0.0.1:43491)
0620 20:21:15.911273 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0620 20:21:15.911278 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:21:15.911586 (+ 308us) server_negotiation.cc:407] Connection header received
0620 20:21:15.911755 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:21:15.911759 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:21:15.911803 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:21:15.911880 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:21:15.912910 (+ 1030us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:21:15.912915 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:21:15.912918 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:21:15.912932 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:21:15.912974 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:21:15.912989 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:21:15.913101 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0620 20:21:15.913226 (+ 125us) server_negotiation.cc:299] Negotiation successful
0620 20:21:15.913271 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260620 20:21:15.913771 6138 log.cc:824] T 00000000000000000000000000000000 P 499f46a5e8d9413aa9b464df6b14e5f5: Log is configured to *not* fsync() on all Append() calls
I20260620 20:21:15.916603 6138 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 499f46a5e8d9413aa9b464df6b14e5f5: No bootstrap required, opened a new log
I20260620 20:21:15.924175 6138 raft_consensus.cc:348] T 00000000000000000000000000000000 P 499f46a5e8d9413aa9b464df6b14e5f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "499f46a5e8d9413aa9b464df6b14e5f5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43491 } }
I20260620 20:21:15.924507 6138 raft_consensus.cc:374] T 00000000000000000000000000000000 P 499f46a5e8d9413aa9b464df6b14e5f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:21:15.924631 6138 raft_consensus.cc:729] T 00000000000000000000000000000000 P 499f46a5e8d9413aa9b464df6b14e5f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 499f46a5e8d9413aa9b464df6b14e5f5, State: Initialized, Role: FOLLOWER
I20260620 20:21:15.925268 6138 consensus_queue.cc:260] T 00000000000000000000000000000000 P 499f46a5e8d9413aa9b464df6b14e5f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "499f46a5e8d9413aa9b464df6b14e5f5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43491 } }
I20260620 20:21:15.925556 6138 raft_consensus.cc:388] T 00000000000000000000000000000000 P 499f46a5e8d9413aa9b464df6b14e5f5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 20:21:15.925714 6138 raft_consensus.cc:482] T 00000000000000000000000000000000 P 499f46a5e8d9413aa9b464df6b14e5f5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 20:21:15.925848 6138 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 499f46a5e8d9413aa9b464df6b14e5f5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:21:15.927977 6138 raft_consensus.cc:504] T 00000000000000000000000000000000 P 499f46a5e8d9413aa9b464df6b14e5f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "499f46a5e8d9413aa9b464df6b14e5f5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43491 } }
I20260620 20:21:15.928413 6138 leader_election.cc:302] T 00000000000000000000000000000000 P 499f46a5e8d9413aa9b464df6b14e5f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 499f46a5e8d9413aa9b464df6b14e5f5; no voters:
I20260620 20:21:15.928953 6138 leader_election.cc:288] T 00000000000000000000000000000000 P 499f46a5e8d9413aa9b464df6b14e5f5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 20:21:15.929121 6148 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 499f46a5e8d9413aa9b464df6b14e5f5 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:21:15.929811 6148 raft_consensus.cc:686] T 00000000000000000000000000000000 P 499f46a5e8d9413aa9b464df6b14e5f5 [term 1 LEADER]: Becoming Leader. State: Replica: 499f46a5e8d9413aa9b464df6b14e5f5, State: Running, Role: LEADER
I20260620 20:21:15.930375 6148 consensus_queue.cc:237] T 00000000000000000000000000000000 P 499f46a5e8d9413aa9b464df6b14e5f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "499f46a5e8d9413aa9b464df6b14e5f5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43491 } }
I20260620 20:21:15.931020 6138 sys_catalog.cc:564] T 00000000000000000000000000000000 P 499f46a5e8d9413aa9b464df6b14e5f5 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:21:15.933125 6150 sys_catalog.cc:455] T 00000000000000000000000000000000 P 499f46a5e8d9413aa9b464df6b14e5f5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 499f46a5e8d9413aa9b464df6b14e5f5. Latest consensus state: current_term: 1 leader_uuid: "499f46a5e8d9413aa9b464df6b14e5f5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "499f46a5e8d9413aa9b464df6b14e5f5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43491 } } }
I20260620 20:21:15.933316 6149 sys_catalog.cc:455] T 00000000000000000000000000000000 P 499f46a5e8d9413aa9b464df6b14e5f5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "499f46a5e8d9413aa9b464df6b14e5f5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "499f46a5e8d9413aa9b464df6b14e5f5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43491 } } }
I20260620 20:21:15.933569 6149 sys_catalog.cc:458] T 00000000000000000000000000000000 P 499f46a5e8d9413aa9b464df6b14e5f5 [sys.catalog]: This master's current role is: LEADER
I20260620 20:21:15.933569 6150 sys_catalog.cc:458] T 00000000000000000000000000000000 P 499f46a5e8d9413aa9b464df6b14e5f5 [sys.catalog]: This master's current role is: LEADER
I20260620 20:21:15.934789 6155 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 20:21:15.940389 6155 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 20:21:15.949329 6155 catalog_manager.cc:1347] Generated new cluster ID: c93c1aba58f04b66aa8c04d60c31b2a4
I20260620 20:21:15.949457 6155 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 20:21:15.981169 6155 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 20:21:15.981828 6155 catalog_manager.cc:1504] Loading token signing keys...
I20260620 20:21:15.990738 6155 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 499f46a5e8d9413aa9b464df6b14e5f5: Generated new TSK 0
I20260620 20:21:15.991786 6155 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 20:21:16.005705 17252 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu pid 6071
W20260620 20:21:16.006861 6133 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.111:38669
W20260620 20:21:16.007668 17252 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu pid 6071: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 20:21:16.007822 17252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu with pid 6071
2026-06-20T20:21:16Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1908 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781986536687152-17252-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 20:21:16 dist-test-slave-2hpg krb5kdc[6169](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 20:21:16 dist-test-slave-2hpg krb5kdc[6169](info): set up 2 sockets
Jun 20 20:21:16 dist-test-slave-2hpg krb5kdc[6169](info): commencing operation
krb5kdc: starting...
W20260620 20:21:17.206563 17252 mini_kdc.cc:121] Time spent starting KDC: real 1.162s user 0.007s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781986536687152-17252-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781986536687152-17252-0/krb5kdc/test-user.keytab.
Jun 20 20:21:17 dist-test-slave-2hpg krb5kdc[6169](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986877, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T20:21:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T20:21:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781986536687152-17252-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781986536687152-17252-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 20:21:17.460943 17252 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
/tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781986536687152-17252-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781986536687152-17252-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781986536687152-17252-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781986536687152-17252-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36533
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781986536687152-17252-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:32829
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36533
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781986536687152-17252-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781986536687152-17252-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781986536687152-17252-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:21:17.800580 6182 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:21:17.800890 6182 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:21:17.801072 6182 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:21:17.808684 6182 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:21:17.808809 6182 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:21:17.808948 6182 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:21:17.809064 6182 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:21:17.809159 6182 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:21:17.819525 6182 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:32829
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781986536687152-17252-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781986536687152-17252-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36533
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781986536687152-17252-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36533
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781986536687152-17252-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781986536687152-17252-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:21:17.820897 6182 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:21:17.822371 6182 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:21:17.835069 6188 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:21:17.835060 6190 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:21:17.835772 6187 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:21:17.836073 6182 server_base.cc:1034] running on GCE node
I20260620 20:21:17.836879 6182 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:21:17.838435 6182 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:21:17.839671 6182 hybrid_clock.cc:648] HybridClock initialized: now 1781986877839621 us; error 62 us; skew 500 ppm
I20260620 20:21:17.840059 6182 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:21:17 dist-test-slave-2hpg krb5kdc[6169](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986877, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:21:17.843362 6182 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 20:21:17.844915 6182 webserver.cc:460] Webserver started at http://127.0.0.1:36015/ using document root <none> and password file <none>
I20260620 20:21:17.845628 6182 fs_manager.cc:359] Metadata directory not provided
I20260620 20:21:17.845791 6182 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:21:17.846149 6182 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:21:17.849220 6182 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781986536687152-17252-0/minicluster-data/master-0/data/instance:
uuid: "4586c026c1864fe690d08fe8d3db8abe"
format_stamp: "Formatted at 2026-06-20 20:21:17 on dist-test-slave-2hpg"
server_key: "7c3045d2d627cb7fc8c6e9f7c4d65809"
server_key_iv: "08c85b1680aff54a10d6eb6949e6cf0d"
server_key_version: "encryptionkey@0"
I20260620 20:21:17.850152 6182 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781986536687152-17252-0/minicluster-data/master-0/wal/instance:
uuid: "4586c026c1864fe690d08fe8d3db8abe"
format_stamp: "Formatted at 2026-06-20 20:21:17 on dist-test-slave-2hpg"
server_key: "7c3045d2d627cb7fc8c6e9f7c4d65809"
server_key_iv: "08c85b1680aff54a10d6eb6949e6cf0d"
server_key_version: "encryptionkey@0"
I20260620 20:21:17.855695 6182 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 20:21:17.859308 6197 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:21:17.860780 6182 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 20:21:17.860980 6182 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781986536687152-17252-0/minicluster-data/master-0/data,/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781986536687152-17252-0/minicluster-data/master-0/wal
uuid: "4586c026c1864fe690d08fe8d3db8abe"
format_stamp: "Formatted at 2026-06-20 20:21:17 on dist-test-slave-2hpg"
server_key: "7c3045d2d627cb7fc8c6e9f7c4d65809"
server_key_iv: "08c85b1680aff54a10d6eb6949e6cf0d"
server_key_version: "encryptionkey@0"
I20260620 20:21:17.861191 6182 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781986536687152-17252-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781986536687152-17252-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781986536687152-17252-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:21:17.875945 6182 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:21:17.879944 6182 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:21:17.880234 6182 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:21:17.899065 6182 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36533
I20260620 20:21:17.899094 6248 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36533 every 8 connection(s)
I20260620 20:21:17.900434 6182 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781986536687152-17252-0/minicluster-data/master-0/data/info.pb
I20260620 20:21:17.904259 6249 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:21:17.905550 17252 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu as pid 6182
I20260620 20:21:17.905733 17252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781986536687152-17252-0/minicluster-data/master-0/wal/instance
I20260620 20:21:17.906320 17252 external_mini_cluster.cc:1428] Setting key 561a6ff8fc0de155e2ecc3ddeefc7223
I20260620 20:21:17.917083 6249 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4586c026c1864fe690d08fe8d3db8abe: Bootstrap starting.
Jun 20 20:21:17 dist-test-slave-2hpg krb5kdc[6169](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986877, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 20:21:17.920506 6249 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4586c026c1864fe690d08fe8d3db8abe: Neither blocks nor log segments found. Creating new log.
I20260620 20:21:17.921530 6249 log.cc:824] T 00000000000000000000000000000000 P 4586c026c1864fe690d08fe8d3db8abe: Log is configured to *not* fsync() on all Append() calls
I20260620 20:21:17.924959 6249 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4586c026c1864fe690d08fe8d3db8abe: No bootstrap required, opened a new log
I20260620 20:21:17.929906 6252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:17.908232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.111:50599 (local address 127.0.0.1:36533)
0620 20:21:17.909071 (+ 839us) server_negotiation.cc:206] Beginning negotiation
0620 20:21:17.909084 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 20:21:17.909126 (+ 42us) server_negotiation.cc:407] Connection header received
0620 20:21:17.910862 (+ 1736us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:21:17.910889 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:21:17.911311 (+ 422us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:21:17.911709 (+ 398us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:21:17.913169 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:17.914598 (+ 1429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:17.915907 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:17.916606 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:17.916795 (+ 189us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:21:17.921201 (+ 4406us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:21:17.921219 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:21:17.921234 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:21:17.921275 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:21:17.925075 (+ 3800us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:21:17.925984 (+ 909us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:21:17.925993 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:21:17.926001 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:21:17.926207 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:21:17.926690 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:21:17.926695 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:21:17.926699 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:21:17.927400 (+ 701us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:21:17.927738 (+ 338us) server_negotiation.cc:1035] Waiting for connection context
0620 20:21:17.928181 (+ 443us) server_negotiation.cc:299] Negotiation successful
0620 20:21:17.928545 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":600,"thread_start_us":390,"threads_started":1}
I20260620 20:21:17.933945 17252 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 20:21:17.934907 6249 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4586c026c1864fe690d08fe8d3db8abe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4586c026c1864fe690d08fe8d3db8abe" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36533 } }
I20260620 20:21:17.935285 6249 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4586c026c1864fe690d08fe8d3db8abe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:21:17.935396 6249 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4586c026c1864fe690d08fe8d3db8abe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4586c026c1864fe690d08fe8d3db8abe, State: Initialized, Role: FOLLOWER
I20260620 20:21:17.935977 6249 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4586c026c1864fe690d08fe8d3db8abe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4586c026c1864fe690d08fe8d3db8abe" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36533 } }
I20260620 20:21:17.936225 6249 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4586c026c1864fe690d08fe8d3db8abe [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 20:21:17.936383 6249 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4586c026c1864fe690d08fe8d3db8abe [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 20:21:17.936555 6249 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4586c026c1864fe690d08fe8d3db8abe [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:21:17.939658 6249 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4586c026c1864fe690d08fe8d3db8abe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4586c026c1864fe690d08fe8d3db8abe" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36533 } }
I20260620 20:21:17.940155 6249 leader_election.cc:302] T 00000000000000000000000000000000 P 4586c026c1864fe690d08fe8d3db8abe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4586c026c1864fe690d08fe8d3db8abe; no voters:
I20260620 20:21:17.940800 6249 leader_election.cc:288] T 00000000000000000000000000000000 P 4586c026c1864fe690d08fe8d3db8abe [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 20:21:17.940902 6259 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4586c026c1864fe690d08fe8d3db8abe [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:21:17.943521 6249 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4586c026c1864fe690d08fe8d3db8abe [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:21:17.944327 6259 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4586c026c1864fe690d08fe8d3db8abe [term 1 LEADER]: Becoming Leader. State: Replica: 4586c026c1864fe690d08fe8d3db8abe, State: Running, Role: LEADER
I20260620 20:21:17.945048 6259 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4586c026c1864fe690d08fe8d3db8abe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4586c026c1864fe690d08fe8d3db8abe" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36533 } }
I20260620 20:21:17.949364 6260 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4586c026c1864fe690d08fe8d3db8abe [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4586c026c1864fe690d08fe8d3db8abe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4586c026c1864fe690d08fe8d3db8abe" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36533 } } }
I20260620 20:21:17.949373 6261 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4586c026c1864fe690d08fe8d3db8abe [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4586c026c1864fe690d08fe8d3db8abe. Latest consensus state: current_term: 1 leader_uuid: "4586c026c1864fe690d08fe8d3db8abe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4586c026c1864fe690d08fe8d3db8abe" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36533 } } }
I20260620 20:21:17.950314 6261 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4586c026c1864fe690d08fe8d3db8abe [sys.catalog]: This master's current role is: LEADER
I20260620 20:21:17.950314 6260 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4586c026c1864fe690d08fe8d3db8abe [sys.catalog]: This master's current role is: LEADER
I20260620 20:21:17.951553 6267 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 20:21:17.958643 6252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:17.938142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.111:54293 (local address 127.0.0.1:36533)
0620 20:21:17.938311 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0620 20:21:17.938315 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:21:17.939113 (+ 798us) server_negotiation.cc:407] Connection header received
0620 20:21:17.939281 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:21:17.939285 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:21:17.939364 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:21:17.939468 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:21:17.941376 (+ 1908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:17.942548 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:17.944061 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:17.944784 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:17.944984 (+ 200us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:21:17.948561 (+ 3577us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:21:17.948575 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:21:17.948583 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:21:17.948633 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:21:17.953477 (+ 4844us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:21:17.954785 (+ 1308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:21:17.954793 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:21:17.954799 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:21:17.955002 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:21:17.955699 (+ 697us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:21:17.955707 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:21:17.955712 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:21:17.956276 (+ 564us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:21:17.956625 (+ 349us) server_negotiation.cc:1035] Waiting for connection context
0620 20:21:17.958095 (+ 1470us) server_negotiation.cc:299] Negotiation successful
0620 20:21:17.958282 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 20:21:17.965994 6267 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 20:21:17.975657 6267 catalog_manager.cc:1347] Generated new cluster ID: 189dace11f7242259f581be11d30ab6e
I20260620 20:21:17.975770 6267 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 20:21:18.014573 6267 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 20:21:18.016630 6267 catalog_manager.cc:1504] Loading token signing keys...
I20260620 20:21:18.034014 6267 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4586c026c1864fe690d08fe8d3db8abe: Generated new TSK 0
I20260620 20:21:18.035244 6267 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 20:21:18.051574 17252 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu pid 6182
I20260620 20:21:18.053268 6244 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:21:18.147143 17252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu with pid 6182
W20260620 20:21:18.164525 6167 connection.cc:535] client connection to 127.0.0.1:36533 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36533): Connection reset by peer (error 104)
2026-06-20T20:21:18Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (2139 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781986536687152-17252-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 20:21:18 dist-test-slave-2hpg krb5kdc[6281](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 20:21:18 dist-test-slave-2hpg krb5kdc[6281](info): set up 2 sockets
Jun 20 20:21:18 dist-test-slave-2hpg krb5kdc[6281](info): commencing operation
krb5kdc: starting...
W20260620 20:21:19.254765 17252 mini_kdc.cc:121] Time spent starting KDC: real 1.070s user 0.000s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781986536687152-17252-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781986536687152-17252-0/krb5kdc/test-user.keytab.
Jun 20 20:21:19 dist-test-slave-2hpg krb5kdc[6281](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986879, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T20:21:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T20:21:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781986536687152-17252-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781986536687152-17252-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 20:21:19.516691 17252 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
/tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781986536687152-17252-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781986536687152-17252-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781986536687152-17252-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781986536687152-17252-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41835
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781986536687152-17252-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46235
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41835
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781986536687152-17252-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781986536687152-17252-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781986536687152-17252-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:21:19.850059 6294 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:21:19.850390 6294 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:21:19.850585 6294 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:21:19.858047 6294 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:21:19.858199 6294 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:21:19.858316 6294 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:21:19.858430 6294 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:21:19.858532 6294 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:21:19.869163 6294 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46235
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781986536687152-17252-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781986536687152-17252-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41835
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781986536687152-17252-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41835
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781986536687152-17252-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781986536687152-17252-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:21:19.870541 6294 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:21:19.871927 6294 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:21:19.884222 6299 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:21:19.884222 6302 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:21:19.884256 6300 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:21:19.885224 6294 server_base.cc:1034] running on GCE node
I20260620 20:21:19.886171 6294 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:21:19.887598 6294 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:21:19.888849 6294 hybrid_clock.cc:648] HybridClock initialized: now 1781986879888809 us; error 44 us; skew 500 ppm
I20260620 20:21:19.889223 6294 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:21:19 dist-test-slave-2hpg krb5kdc[6281](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986879, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:21:19.896425 6294 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 20:21:19.898170 6294 webserver.cc:460] Webserver started at http://127.0.0.1:42887/ using document root <none> and password file <none>
I20260620 20:21:19.898926 6294 fs_manager.cc:359] Metadata directory not provided
I20260620 20:21:19.899096 6294 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:21:19.899426 6294 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:21:19.902735 6294 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781986536687152-17252-0/minicluster-data/master-0/data/instance:
uuid: "202498773885460e991a6e0e59ae1b3f"
format_stamp: "Formatted at 2026-06-20 20:21:19 on dist-test-slave-2hpg"
server_key: "64011c58d7c49e97eb286bbd6731b534"
server_key_iv: "5751c9393d018b1722ee1b3647e97146"
server_key_version: "encryptionkey@0"
I20260620 20:21:19.903590 6294 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781986536687152-17252-0/minicluster-data/master-0/wal/instance:
uuid: "202498773885460e991a6e0e59ae1b3f"
format_stamp: "Formatted at 2026-06-20 20:21:19 on dist-test-slave-2hpg"
server_key: "64011c58d7c49e97eb286bbd6731b534"
server_key_iv: "5751c9393d018b1722ee1b3647e97146"
server_key_version: "encryptionkey@0"
I20260620 20:21:19.909644 6294 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260620 20:21:19.913334 6309 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:21:19.914764 6294 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 20:21:19.914959 6294 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781986536687152-17252-0/minicluster-data/master-0/data,/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781986536687152-17252-0/minicluster-data/master-0/wal
uuid: "202498773885460e991a6e0e59ae1b3f"
format_stamp: "Formatted at 2026-06-20 20:21:19 on dist-test-slave-2hpg"
server_key: "64011c58d7c49e97eb286bbd6731b534"
server_key_iv: "5751c9393d018b1722ee1b3647e97146"
server_key_version: "encryptionkey@0"
I20260620 20:21:19.915171 6294 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781986536687152-17252-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781986536687152-17252-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781986536687152-17252-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:21:19.927505 6294 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:21:19.931964 6294 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:21:19.932297 6294 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:21:19.951709 6294 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41835
I20260620 20:21:19.951745 6360 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41835 every 8 connection(s)
I20260620 20:21:19.953096 6294 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781986536687152-17252-0/minicluster-data/master-0/data/info.pb
I20260620 20:21:19.956821 6361 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:21:19.960753 17252 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu as pid 6294
I20260620 20:21:19.960922 17252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781986536687152-17252-0/minicluster-data/master-0/wal/instance
I20260620 20:21:19.961331 17252 external_mini_cluster.cc:1428] Setting key 4e2b3672fdeeb4bdc10241974d1b9f1e
I20260620 20:21:19.970121 6361 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 202498773885460e991a6e0e59ae1b3f: Bootstrap starting.
I20260620 20:21:19.974375 6361 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 202498773885460e991a6e0e59ae1b3f: Neither blocks nor log segments found. Creating new log.
I20260620 20:21:19.975395 6364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:19.963086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.111:46525 (local address 127.0.0.1:41835)
0620 20:21:19.963869 (+ 783us) server_negotiation.cc:206] Beginning negotiation
0620 20:21:19.963880 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 20:21:19.963919 (+ 39us) server_negotiation.cc:407] Connection header received
0620 20:21:19.965551 (+ 1632us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:21:19.965574 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:21:19.966047 (+ 473us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:21:19.966377 (+ 330us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:21:19.967078 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:19.968436 (+ 1358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:19.970512 (+ 2076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:19.971245 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:19.971378 (+ 133us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:21:19.972864 (+ 1486us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:21:19.972872 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:21:19.972887 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:21:19.973467 (+ 580us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:21:19.973548 (+ 81us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:21:19.973574 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:21:19.973699 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0620 20:21:19.974095 (+ 396us) server_negotiation.cc:299] Negotiation successful
0620 20:21:19.974304 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":572,"thread_start_us":356,"threads_started":1}
I20260620 20:21:19.975666 6361 log.cc:824] T 00000000000000000000000000000000 P 202498773885460e991a6e0e59ae1b3f: Log is configured to *not* fsync() on all Append() calls
I20260620 20:21:19.979171 6361 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 202498773885460e991a6e0e59ae1b3f: No bootstrap required, opened a new log
I20260620 20:21:19.979514 17252 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 20:21:19.988920 6361 raft_consensus.cc:348] T 00000000000000000000000000000000 P 202498773885460e991a6e0e59ae1b3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "202498773885460e991a6e0e59ae1b3f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41835 } }
I20260620 20:21:19.989506 6361 raft_consensus.cc:374] T 00000000000000000000000000000000 P 202498773885460e991a6e0e59ae1b3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:21:19.989638 6361 raft_consensus.cc:729] T 00000000000000000000000000000000 P 202498773885460e991a6e0e59ae1b3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 202498773885460e991a6e0e59ae1b3f, State: Initialized, Role: FOLLOWER
I20260620 20:21:19.990434 6361 consensus_queue.cc:260] T 00000000000000000000000000000000 P 202498773885460e991a6e0e59ae1b3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "202498773885460e991a6e0e59ae1b3f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41835 } }
I20260620 20:21:19.990762 6361 raft_consensus.cc:388] T 00000000000000000000000000000000 P 202498773885460e991a6e0e59ae1b3f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 20:21:19.990994 6361 raft_consensus.cc:482] T 00000000000000000000000000000000 P 202498773885460e991a6e0e59ae1b3f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 20:21:19.991238 6364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:19.984047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.111:33801 (local address 127.0.0.1:41835)
0620 20:21:19.984255 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0620 20:21:19.984267 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 20:21:19.984571 (+ 304us) server_negotiation.cc:407] Connection header received
0620 20:21:19.984832 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:21:19.984838 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:21:19.984928 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:21:19.985055 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:21:19.985542 (+ 487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:19.986876 (+ 1334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:19.988217 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:19.988919 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:19.989061 (+ 142us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:21:19.990286 (+ 1225us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:21:19.990293 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:21:19.990298 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:21:19.990320 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:21:19.990394 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:21:19.990426 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:21:19.990578 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0620 20:21:19.990762 (+ 184us) server_negotiation.cc:299] Negotiation successful
0620 20:21:19.990823 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260620 20:21:19.991235 6361 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 202498773885460e991a6e0e59ae1b3f [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:21:19.994218 6361 raft_consensus.cc:504] T 00000000000000000000000000000000 P 202498773885460e991a6e0e59ae1b3f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "202498773885460e991a6e0e59ae1b3f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41835 } }
I20260620 20:21:19.994915 6361 leader_election.cc:302] T 00000000000000000000000000000000 P 202498773885460e991a6e0e59ae1b3f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 202498773885460e991a6e0e59ae1b3f; no voters:
I20260620 20:21:19.995626 6361 leader_election.cc:288] T 00000000000000000000000000000000 P 202498773885460e991a6e0e59ae1b3f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 20:21:19.995803 6371 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 202498773885460e991a6e0e59ae1b3f [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:21:19.996539 6371 raft_consensus.cc:686] T 00000000000000000000000000000000 P 202498773885460e991a6e0e59ae1b3f [term 1 LEADER]: Becoming Leader. State: Replica: 202498773885460e991a6e0e59ae1b3f, State: Running, Role: LEADER
I20260620 20:21:19.997153 6371 consensus_queue.cc:237] T 00000000000000000000000000000000 P 202498773885460e991a6e0e59ae1b3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "202498773885460e991a6e0e59ae1b3f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41835 } }
I20260620 20:21:19.998064 6361 sys_catalog.cc:564] T 00000000000000000000000000000000 P 202498773885460e991a6e0e59ae1b3f [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:21:20.000397 6372 sys_catalog.cc:455] T 00000000000000000000000000000000 P 202498773885460e991a6e0e59ae1b3f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "202498773885460e991a6e0e59ae1b3f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "202498773885460e991a6e0e59ae1b3f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41835 } } }
I20260620 20:21:20.000929 6372 sys_catalog.cc:458] T 00000000000000000000000000000000 P 202498773885460e991a6e0e59ae1b3f [sys.catalog]: This master's current role is: LEADER
I20260620 20:21:20.000699 6373 sys_catalog.cc:455] T 00000000000000000000000000000000 P 202498773885460e991a6e0e59ae1b3f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 202498773885460e991a6e0e59ae1b3f. Latest consensus state: current_term: 1 leader_uuid: "202498773885460e991a6e0e59ae1b3f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "202498773885460e991a6e0e59ae1b3f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41835 } } }
I20260620 20:21:20.001096 6373 sys_catalog.cc:458] T 00000000000000000000000000000000 P 202498773885460e991a6e0e59ae1b3f [sys.catalog]: This master's current role is: LEADER
I20260620 20:21:20.003408 6377 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 20:21:20.009107 6377 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 20:21:20.020308 6377 catalog_manager.cc:1347] Generated new cluster ID: e2f060a7ce204ec2ae85e7c05fb1bafe
I20260620 20:21:20.020449 6377 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 20:21:20.035295 6377 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 20:21:20.036715 6377 catalog_manager.cc:1504] Loading token signing keys...
I20260620 20:21:20.044158 6377 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 202498773885460e991a6e0e59ae1b3f: Generated new TSK 0
I20260620 20:21:20.045199 6377 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 20:21:20.060835 17252 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu pid 6294
W20260620 20:21:20.062212 6356 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.111:46525
W20260620 20:21:20.062987 17252 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu pid 6294: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 20:21:20.063145 17252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu with pid 6294
W20260620 20:21:20.077111 6279 connection.cc:535] client connection to 127.0.0.1:41835 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41835): Connection reset by peer (error 104)
2026-06-20T20:21:20Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1911 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781986536687152-17252-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 20:21:20 dist-test-slave-2hpg krb5kdc[6392](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 20:21:20 dist-test-slave-2hpg krb5kdc[6392](info): set up 2 sockets
Jun 20 20:21:20 dist-test-slave-2hpg krb5kdc[6392](info): commencing operation
krb5kdc: starting...
W20260620 20:21:21.166816 17252 mini_kdc.cc:121] Time spent starting KDC: real 1.071s user 0.000s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781986536687152-17252-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781986536687152-17252-0/krb5kdc/test-user.keytab.
Jun 20 20:21:21 dist-test-slave-2hpg krb5kdc[6392](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986881, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T20:21:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T20:21:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781986536687152-17252-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781986536687152-17252-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 20:21:21.428932 17252 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
/tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781986536687152-17252-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781986536687152-17252-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781986536687152-17252-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781986536687152-17252-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45937
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781986536687152-17252-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44295
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45937
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781986536687152-17252-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781986536687152-17252-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781986536687152-17252-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:21:21.763543 6405 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:21:21.763816 6405 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:21:21.764011 6405 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:21:21.771330 6405 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:21:21.771457 6405 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:21:21.771560 6405 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:21:21.771680 6405 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:21:21.771757 6405 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:21:21.781507 6405 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44295
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781986536687152-17252-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781986536687152-17252-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45937
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781986536687152-17252-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45937
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781986536687152-17252-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781986536687152-17252-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:21:21.782814 6405 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:21:21.784088 6405 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:21:21.795523 6411 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:21:21.795578 6410 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:21:21.795609 6413 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:21:21.796268 6405 server_base.cc:1034] running on GCE node
I20260620 20:21:21.797052 6405 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:21:21.798466 6405 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:21:21.799741 6405 hybrid_clock.cc:648] HybridClock initialized: now 1781986881799679 us; error 48 us; skew 500 ppm
I20260620 20:21:21.800107 6405 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:21:21 dist-test-slave-2hpg krb5kdc[6392](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986881, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:21:21.803670 6405 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 20:21:21.805066 6405 webserver.cc:460] Webserver started at http://127.0.0.1:38517/ using document root <none> and password file <none>
I20260620 20:21:21.805794 6405 fs_manager.cc:359] Metadata directory not provided
I20260620 20:21:21.805955 6405 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:21:21.806273 6405 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:21:21.809273 6405 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781986536687152-17252-0/minicluster-data/master-0/data/instance:
uuid: "8febb49422ed44c6baa4c7e2428bc991"
format_stamp: "Formatted at 2026-06-20 20:21:21 on dist-test-slave-2hpg"
server_key: "75283a21d465803e56ef088351b76ac1"
server_key_iv: "deabffddefbb286ba520e13f3c433347"
server_key_version: "encryptionkey@0"
I20260620 20:21:21.810107 6405 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781986536687152-17252-0/minicluster-data/master-0/wal/instance:
uuid: "8febb49422ed44c6baa4c7e2428bc991"
format_stamp: "Formatted at 2026-06-20 20:21:21 on dist-test-slave-2hpg"
server_key: "75283a21d465803e56ef088351b76ac1"
server_key_iv: "deabffddefbb286ba520e13f3c433347"
server_key_version: "encryptionkey@0"
I20260620 20:21:21.815266 6405 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.002s
I20260620 20:21:21.818717 6420 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:21:21.820000 6405 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260620 20:21:21.820191 6405 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781986536687152-17252-0/minicluster-data/master-0/data,/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781986536687152-17252-0/minicluster-data/master-0/wal
uuid: "8febb49422ed44c6baa4c7e2428bc991"
format_stamp: "Formatted at 2026-06-20 20:21:21 on dist-test-slave-2hpg"
server_key: "75283a21d465803e56ef088351b76ac1"
server_key_iv: "deabffddefbb286ba520e13f3c433347"
server_key_version: "encryptionkey@0"
I20260620 20:21:21.820399 6405 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781986536687152-17252-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781986536687152-17252-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781986536687152-17252-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:21:21.826275 6405 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:21:21.830319 6405 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:21:21.830615 6405 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:21:21.850409 6405 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45937
I20260620 20:21:21.850512 6471 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45937 every 8 connection(s)
I20260620 20:21:21.851881 6405 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781986536687152-17252-0/minicluster-data/master-0/data/info.pb
I20260620 20:21:21.852890 17252 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu as pid 6405
I20260620 20:21:21.853081 17252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781986536687152-17252-0/minicluster-data/master-0/wal/instance
I20260620 20:21:21.853535 17252 external_mini_cluster.cc:1428] Setting key 5f02100bfe4faa147cc522a97b9d40eb
I20260620 20:21:21.856365 6472 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:21:21.862443 6475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:21.855431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.111:50003 (local address 127.0.0.1:45937)
0620 20:21:21.856203 (+ 772us) server_negotiation.cc:206] Beginning negotiation
0620 20:21:21.856219 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 20:21:21.856267 (+ 48us) server_negotiation.cc:407] Connection header received
0620 20:21:21.858637 (+ 2370us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:21:21.858664 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:21:21.859072 (+ 408us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:21:21.859502 (+ 430us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:21:21.860505 (+ 1003us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:21:21.860513 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:21:21.860529 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:21:21.860545 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:21:21.860642 (+ 97us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:21:21.860668 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:21:21.860890 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0620 20:21:21.861032 (+ 142us) server_negotiation.cc:299] Negotiation successful
0620 20:21:21.861228 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":530,"thread_start_us":299,"threads_started":1}
I20260620 20:21:21.866008 17252 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 20:21:21.868877 6472 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8febb49422ed44c6baa4c7e2428bc991: Bootstrap starting.
I20260620 20:21:21.872960 6472 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8febb49422ed44c6baa4c7e2428bc991: Neither blocks nor log segments found. Creating new log.
I20260620 20:21:21.873112 6475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:21.870239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.111:55589 (local address 127.0.0.1:45937)
0620 20:21:21.870527 (+ 288us) server_negotiation.cc:206] Beginning negotiation
0620 20:21:21.870539 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 20:21:21.871044 (+ 505us) server_negotiation.cc:407] Connection header received
0620 20:21:21.871202 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:21:21.871205 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:21:21.871251 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:21:21.871327 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:21:21.872313 (+ 986us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:21:21.872317 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:21:21.872320 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:21:21.872333 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:21:21.872368 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:21:21.872382 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:21:21.872467 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0620 20:21:21.872643 (+ 176us) server_negotiation.cc:299] Negotiation successful
0620 20:21:21.872695 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260620 20:21:21.874174 6472 log.cc:824] T 00000000000000000000000000000000 P 8febb49422ed44c6baa4c7e2428bc991: Log is configured to *not* fsync() on all Append() calls
I20260620 20:21:21.876753 6472 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8febb49422ed44c6baa4c7e2428bc991: No bootstrap required, opened a new log
I20260620 20:21:21.883876 6472 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8febb49422ed44c6baa4c7e2428bc991 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8febb49422ed44c6baa4c7e2428bc991" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45937 } }
I20260620 20:21:21.884240 6472 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8febb49422ed44c6baa4c7e2428bc991 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:21:21.884344 6472 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8febb49422ed44c6baa4c7e2428bc991 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8febb49422ed44c6baa4c7e2428bc991, State: Initialized, Role: FOLLOWER
I20260620 20:21:21.884886 6472 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8febb49422ed44c6baa4c7e2428bc991 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8febb49422ed44c6baa4c7e2428bc991" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45937 } }
I20260620 20:21:21.885120 6472 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8febb49422ed44c6baa4c7e2428bc991 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 20:21:21.885263 6472 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8febb49422ed44c6baa4c7e2428bc991 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 20:21:21.885434 6472 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8febb49422ed44c6baa4c7e2428bc991 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:21:21.887598 6472 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8febb49422ed44c6baa4c7e2428bc991 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8febb49422ed44c6baa4c7e2428bc991" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45937 } }
I20260620 20:21:21.888085 6472 leader_election.cc:302] T 00000000000000000000000000000000 P 8febb49422ed44c6baa4c7e2428bc991 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8febb49422ed44c6baa4c7e2428bc991; no voters:
I20260620 20:21:21.888667 6472 leader_election.cc:288] T 00000000000000000000000000000000 P 8febb49422ed44c6baa4c7e2428bc991 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 20:21:21.888795 6482 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8febb49422ed44c6baa4c7e2428bc991 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:21:21.889446 6482 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8febb49422ed44c6baa4c7e2428bc991 [term 1 LEADER]: Becoming Leader. State: Replica: 8febb49422ed44c6baa4c7e2428bc991, State: Running, Role: LEADER
I20260620 20:21:21.889997 6482 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8febb49422ed44c6baa4c7e2428bc991 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8febb49422ed44c6baa4c7e2428bc991" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45937 } }
I20260620 20:21:21.890704 6472 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8febb49422ed44c6baa4c7e2428bc991 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:21:21.892732 6483 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8febb49422ed44c6baa4c7e2428bc991 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8febb49422ed44c6baa4c7e2428bc991" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8febb49422ed44c6baa4c7e2428bc991" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45937 } } }
I20260620 20:21:21.893211 6483 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8febb49422ed44c6baa4c7e2428bc991 [sys.catalog]: This master's current role is: LEADER
I20260620 20:21:21.892760 6484 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8febb49422ed44c6baa4c7e2428bc991 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8febb49422ed44c6baa4c7e2428bc991. Latest consensus state: current_term: 1 leader_uuid: "8febb49422ed44c6baa4c7e2428bc991" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8febb49422ed44c6baa4c7e2428bc991" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45937 } } }
I20260620 20:21:21.894580 6484 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8febb49422ed44c6baa4c7e2428bc991 [sys.catalog]: This master's current role is: LEADER
I20260620 20:21:21.896071 6487 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 20:21:21.901594 6487 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 20:21:21.911434 6487 catalog_manager.cc:1347] Generated new cluster ID: 07329bba93e14bb69a64c08db8838af7
I20260620 20:21:21.911598 6487 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 20:21:21.930545 6487 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 20:21:21.931069 6487 catalog_manager.cc:1504] Loading token signing keys...
I20260620 20:21:21.940858 6487 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8febb49422ed44c6baa4c7e2428bc991: Generated new TSK 0
I20260620 20:21:21.941885 6487 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 20:21:21.960201 17252 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu pid 6405
W20260620 20:21:21.961488 6467 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.111:50003
W20260620 20:21:21.962311 17252 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu pid 6405: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 20:21:21.962472 17252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu with pid 6405
2026-06-20T20:21:21Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1902 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781986536687152-17252-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 20:21:22 dist-test-slave-2hpg krb5kdc[6503](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 20:21:22 dist-test-slave-2hpg krb5kdc[6503](info): set up 2 sockets
Jun 20 20:21:22 dist-test-slave-2hpg krb5kdc[6503](info): commencing operation
krb5kdc: starting...
W20260620 20:21:23.054688 17252 mini_kdc.cc:121] Time spent starting KDC: real 1.056s user 0.000s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781986536687152-17252-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781986536687152-17252-0/krb5kdc/test-user.keytab.
Jun 20 20:21:23 dist-test-slave-2hpg krb5kdc[6503](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986883, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T20:21:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T20:21:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.217.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.217.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.217.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.62.keytab.
Entry for principal kudu/127.16.217.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.62.keytab.
I20260620 20:21:23.313582 17252 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
/tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781986536687152-17252-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781986536687152-17252-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781986536687152-17252-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781986536687152-17252-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.217.62:38685
--webserver_interface=127.16.217.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.62.keytab
--principal=kudu/127.16.217.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.217.20:42211
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.217.62:38685
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781986536687152-17252-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781986536687152-17252-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781986536687152-17252-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:21:23.646437 6516 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:21:23.646814 6516 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:21:23.646909 6516 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:21:23.654093 6516 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:21:23.654208 6516 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:21:23.654258 6516 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:21:23.654320 6516 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:21:23.654364 6516 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:21:23.664211 6516 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.217.20:42211
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781986536687152-17252-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781986536687152-17252-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.217.62:38685
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.62.keytab
--principal=kudu/127.16.217.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.217.62:38685
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781986536687152-17252-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.217.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781986536687152-17252-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:21:23.665427 6516 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:21:23.666692 6516 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:21:23.678753 6521 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:21:23.678804 6524 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:21:23.678846 6522 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:21:23.679812 6516 server_base.cc:1034] running on GCE node
I20260620 20:21:23.680500 6516 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:21:23.681818 6516 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:21:23.683077 6516 hybrid_clock.cc:648] HybridClock initialized: now 1781986883683007 us; error 51 us; skew 500 ppm
I20260620 20:21:23.683418 6516 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:21:23 dist-test-slave-2hpg krb5kdc[6503](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986883, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.217.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:21:23.686491 6516 init.cc:377] Logged in from keytab as kudu/127.16.217.62@KRBTEST.COM (short username kudu)
I20260620 20:21:23.688009 6516 webserver.cc:460] Webserver started at http://127.16.217.62:42297/ using document root <none> and password file <none>
I20260620 20:21:23.688688 6516 fs_manager.cc:359] Metadata directory not provided
I20260620 20:21:23.688818 6516 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:21:23.689122 6516 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:21:23.692030 6516 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781986536687152-17252-0/minicluster-data/master-0/data/instance:
uuid: "ceddb998315e45c19eed6e8d52637889"
format_stamp: "Formatted at 2026-06-20 20:21:23 on dist-test-slave-2hpg"
server_key: "f617a776dbc45b5736bfdb3b21c99ac6"
server_key_iv: "72cfd36eab9c9b20769ef39f4fd89f08"
server_key_version: "encryptionkey@0"
I20260620 20:21:23.692785 6516 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781986536687152-17252-0/minicluster-data/master-0/wal/instance:
uuid: "ceddb998315e45c19eed6e8d52637889"
format_stamp: "Formatted at 2026-06-20 20:21:23 on dist-test-slave-2hpg"
server_key: "f617a776dbc45b5736bfdb3b21c99ac6"
server_key_iv: "72cfd36eab9c9b20769ef39f4fd89f08"
server_key_version: "encryptionkey@0"
I20260620 20:21:23.697826 6516 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260620 20:21:23.701308 6531 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:21:23.702685 6516 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 20:21:23.702867 6516 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781986536687152-17252-0/minicluster-data/master-0/data,/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781986536687152-17252-0/minicluster-data/master-0/wal
uuid: "ceddb998315e45c19eed6e8d52637889"
format_stamp: "Formatted at 2026-06-20 20:21:23 on dist-test-slave-2hpg"
server_key: "f617a776dbc45b5736bfdb3b21c99ac6"
server_key_iv: "72cfd36eab9c9b20769ef39f4fd89f08"
server_key_version: "encryptionkey@0"
I20260620 20:21:23.703068 6516 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781986536687152-17252-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781986536687152-17252-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781986536687152-17252-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:21:23.724370 6516 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:21:23.728328 6516 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:21:23.728606 6516 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:21:23.748112 6516 rpc_server.cc:307] RPC server started. Bound to: 127.16.217.62:38685
I20260620 20:21:23.748147 6582 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.217.62:38685 every 8 connection(s)
I20260620 20:21:23.749480 6516 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781986536687152-17252-0/minicluster-data/master-0/data/info.pb
I20260620 20:21:23.753137 6583 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:21:23.758814 17252 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu as pid 6516
I20260620 20:21:23.759006 17252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781986536687152-17252-0/minicluster-data/master-0/wal/instance
I20260620 20:21:23.759420 17252 external_mini_cluster.cc:1428] Setting key dc3d8d5cf1ee717d1c95f1110be3b0ec
I20260620 20:21:23.765828 6583 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ceddb998315e45c19eed6e8d52637889: Bootstrap starting.
I20260620 20:21:23.769702 6583 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ceddb998315e45c19eed6e8d52637889: Neither blocks nor log segments found. Creating new log.
I20260620 20:21:23.770826 6583 log.cc:824] T 00000000000000000000000000000000 P ceddb998315e45c19eed6e8d52637889: Log is configured to *not* fsync() on all Append() calls
I20260620 20:21:23.773859 6583 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ceddb998315e45c19eed6e8d52637889: No bootstrap required, opened a new log
Jun 20 20:21:23 dist-test-slave-2hpg krb5kdc[6503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986883, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.217.62@KRBTEST.COM
I20260620 20:21:23.784214 6583 raft_consensus.cc:348] T 00000000000000000000000000000000 P ceddb998315e45c19eed6e8d52637889 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ceddb998315e45c19eed6e8d52637889" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 38685 } }
I20260620 20:21:23.784673 6583 raft_consensus.cc:374] T 00000000000000000000000000000000 P ceddb998315e45c19eed6e8d52637889 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:21:23.784780 6583 raft_consensus.cc:729] T 00000000000000000000000000000000 P ceddb998315e45c19eed6e8d52637889 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ceddb998315e45c19eed6e8d52637889, State: Initialized, Role: FOLLOWER
I20260620 20:21:23.785470 6583 consensus_queue.cc:260] T 00000000000000000000000000000000 P ceddb998315e45c19eed6e8d52637889 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ceddb998315e45c19eed6e8d52637889" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 38685 } }
I20260620 20:21:23.785755 6586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:23.761130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33709 (local address 127.16.217.62:38685)
0620 20:21:23.762912 (+ 1782us) server_negotiation.cc:206] Beginning negotiation
0620 20:21:23.762929 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 20:21:23.762981 (+ 52us) server_negotiation.cc:407] Connection header received
0620 20:21:23.765432 (+ 2451us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:21:23.765464 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:21:23.766030 (+ 566us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:21:23.766487 (+ 457us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:21:23.767857 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:23.769836 (+ 1979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:23.771270 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:23.772139 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:23.772328 (+ 189us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:21:23.776587 (+ 4259us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:21:23.776598 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:21:23.776617 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:21:23.776673 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:21:23.780835 (+ 4162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:21:23.781735 (+ 900us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:21:23.781749 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:21:23.781758 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:21:23.782005 (+ 247us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:21:23.782496 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:21:23.782501 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:21:23.782505 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:21:23.783243 (+ 738us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:21:23.783543 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0620 20:21:23.784008 (+ 465us) server_negotiation.cc:299] Negotiation successful
0620 20:21:23.784344 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1502,"thread_start_us":1225,"threads_started":1}
I20260620 20:21:23.785763 6583 raft_consensus.cc:388] T 00000000000000000000000000000000 P ceddb998315e45c19eed6e8d52637889 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 20:21:23.785945 6583 raft_consensus.cc:482] T 00000000000000000000000000000000 P ceddb998315e45c19eed6e8d52637889 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 20:21:23.786163 6583 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ceddb998315e45c19eed6e8d52637889 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:21:23.789000 6583 raft_consensus.cc:504] T 00000000000000000000000000000000 P ceddb998315e45c19eed6e8d52637889 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ceddb998315e45c19eed6e8d52637889" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 38685 } }
I20260620 20:21:23.789568 6583 leader_election.cc:302] T 00000000000000000000000000000000 P ceddb998315e45c19eed6e8d52637889 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ceddb998315e45c19eed6e8d52637889; no voters:
I20260620 20:21:23.789829 17252 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 20:21:23.790356 6583 leader_election.cc:288] T 00000000000000000000000000000000 P ceddb998315e45c19eed6e8d52637889 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 20:21:23.790459 6588 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ceddb998315e45c19eed6e8d52637889 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:21:23.792811 6583 sys_catalog.cc:564] T 00000000000000000000000000000000 P ceddb998315e45c19eed6e8d52637889 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:21:23.795796 6588 raft_consensus.cc:686] T 00000000000000000000000000000000 P ceddb998315e45c19eed6e8d52637889 [term 1 LEADER]: Becoming Leader. State: Replica: ceddb998315e45c19eed6e8d52637889, State: Running, Role: LEADER
I20260620 20:21:23.796631 6588 consensus_queue.cc:237] T 00000000000000000000000000000000 P ceddb998315e45c19eed6e8d52637889 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ceddb998315e45c19eed6e8d52637889" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 38685 } }
I20260620 20:21:23.803740 6590 sys_catalog.cc:455] T 00000000000000000000000000000000 P ceddb998315e45c19eed6e8d52637889 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ceddb998315e45c19eed6e8d52637889. Latest consensus state: current_term: 1 leader_uuid: "ceddb998315e45c19eed6e8d52637889" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ceddb998315e45c19eed6e8d52637889" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 38685 } } }
I20260620 20:21:23.804236 6590 sys_catalog.cc:458] T 00000000000000000000000000000000 P ceddb998315e45c19eed6e8d52637889 [sys.catalog]: This master's current role is: LEADER
I20260620 20:21:23.804886 6589 sys_catalog.cc:455] T 00000000000000000000000000000000 P ceddb998315e45c19eed6e8d52637889 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ceddb998315e45c19eed6e8d52637889" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ceddb998315e45c19eed6e8d52637889" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 38685 } } }
I20260620 20:21:23.806128 6589 sys_catalog.cc:458] T 00000000000000000000000000000000 P ceddb998315e45c19eed6e8d52637889 [sys.catalog]: This master's current role is: LEADER
I20260620 20:21:23.809562 6605 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 20:21:23.816967 6605 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 20:21:23.824373 6586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:23.798899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52537 (local address 127.16.217.62:38685)
0620 20:21:23.799100 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0620 20:21:23.799110 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 20:21:23.805762 (+ 6652us) server_negotiation.cc:407] Connection header received
0620 20:21:23.806042 (+ 280us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:21:23.806050 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:21:23.806141 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:21:23.806241 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:21:23.808730 (+ 2489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:23.810631 (+ 1901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:23.812934 (+ 2303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:23.813804 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:23.814656 (+ 852us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:21:23.817331 (+ 2675us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:21:23.817338 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:21:23.817343 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:21:23.817384 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:21:23.821329 (+ 3945us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:21:23.822131 (+ 802us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:21:23.822136 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:21:23.822140 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:21:23.822271 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:21:23.822745 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:21:23.822750 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:21:23.822753 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:21:23.823099 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:21:23.823397 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0620 20:21:23.823777 (+ 380us) server_negotiation.cc:299] Negotiation successful
0620 20:21:23.823955 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 20:21:23.826922 6605 catalog_manager.cc:1347] Generated new cluster ID: 10c2c6ed48fc4c2fb39b80db688c60af
I20260620 20:21:23.827034 6605 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 20:21:23.854444 6605 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 20:21:23.856101 6605 catalog_manager.cc:1504] Loading token signing keys...
I20260620 20:21:23.871106 6605 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ceddb998315e45c19eed6e8d52637889: Generated new TSK 0
I20260620 20:21:23.872087 6605 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 20:21:23.878196 17252 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu pid 6516
I20260620 20:21:23.879208 6578 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:21:23.971410 17252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu with pid 6516
W20260620 20:21:23.989012 6501 connection.cc:535] client connection to 127.16.217.62:38685 recv error: Network error: failed to read from TLS socket (remote: 127.16.217.62:38685): Connection reset by peer (error 104)
2026-06-20T20:21:23Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (2009 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781986536687152-17252-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 20:21:24 dist-test-slave-2hpg krb5kdc[6616](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 20:21:24 dist-test-slave-2hpg krb5kdc[6616](info): set up 2 sockets
Jun 20 20:21:24 dist-test-slave-2hpg krb5kdc[6616](info): commencing operation
krb5kdc: starting...
W20260620 20:21:25.118628 17252 mini_kdc.cc:121] Time spent starting KDC: real 1.111s user 0.000s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781986536687152-17252-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781986536687152-17252-0/krb5kdc/test-user.keytab.
Jun 20 20:21:25 dist-test-slave-2hpg krb5kdc[6616](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986885, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T20:21:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T20:21:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.217.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.217.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.217.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.62.keytab.
Entry for principal kudu/127.16.217.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.62.keytab.
I20260620 20:21:25.377964 17252 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
/tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781986536687152-17252-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781986536687152-17252-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781986536687152-17252-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781986536687152-17252-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.217.62:44297
--webserver_interface=127.16.217.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.62.keytab
--principal=kudu/127.16.217.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.217.20:35729
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.217.62:44297
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781986536687152-17252-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781986536687152-17252-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781986536687152-17252-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:21:25.714949 6629 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:21:25.715250 6629 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:21:25.715456 6629 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:21:25.722908 6629 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:21:25.723018 6629 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:21:25.723134 6629 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:21:25.723269 6629 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:21:25.723358 6629 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:21:25.733381 6629 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.217.20:35729
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781986536687152-17252-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781986536687152-17252-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.217.62:44297
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.62.keytab
--principal=kudu/127.16.217.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.217.62:44297
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781986536687152-17252-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.217.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781986536687152-17252-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:21:25.734695 6629 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:21:25.735994 6629 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:21:25.748065 6634 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:21:25.748065 6637 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:21:25.748095 6635 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:21:25.748816 6629 server_base.cc:1034] running on GCE node
I20260620 20:21:25.749594 6629 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:21:25.751001 6629 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:21:25.752247 6629 hybrid_clock.cc:648] HybridClock initialized: now 1781986885752198 us; error 45 us; skew 500 ppm
I20260620 20:21:25.752633 6629 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:21:25 dist-test-slave-2hpg krb5kdc[6616](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986885, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.217.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:21:25.760313 6629 init.cc:377] Logged in from keytab as kudu/127.16.217.62@KRBTEST.COM (short username kudu)
I20260620 20:21:25.762086 6629 webserver.cc:460] Webserver started at http://127.16.217.62:37809/ using document root <none> and password file <none>
I20260620 20:21:25.762837 6629 fs_manager.cc:359] Metadata directory not provided
I20260620 20:21:25.762993 6629 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:21:25.763304 6629 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:21:25.766459 6629 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781986536687152-17252-0/minicluster-data/master-0/data/instance:
uuid: "5ce943b945594c419c213b27fb3c3b9b"
format_stamp: "Formatted at 2026-06-20 20:21:25 on dist-test-slave-2hpg"
server_key: "430a2860111d77137ae4f7c39ebfffb5"
server_key_iv: "81b56228893177b9d7a0992b8aa042fa"
server_key_version: "encryptionkey@0"
I20260620 20:21:25.767247 6629 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781986536687152-17252-0/minicluster-data/master-0/wal/instance:
uuid: "5ce943b945594c419c213b27fb3c3b9b"
format_stamp: "Formatted at 2026-06-20 20:21:25 on dist-test-slave-2hpg"
server_key: "430a2860111d77137ae4f7c39ebfffb5"
server_key_iv: "81b56228893177b9d7a0992b8aa042fa"
server_key_version: "encryptionkey@0"
I20260620 20:21:25.772562 6629 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 20:21:25.776105 6644 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:21:25.777415 6629 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 20:21:25.777622 6629 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781986536687152-17252-0/minicluster-data/master-0/data,/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781986536687152-17252-0/minicluster-data/master-0/wal
uuid: "5ce943b945594c419c213b27fb3c3b9b"
format_stamp: "Formatted at 2026-06-20 20:21:25 on dist-test-slave-2hpg"
server_key: "430a2860111d77137ae4f7c39ebfffb5"
server_key_iv: "81b56228893177b9d7a0992b8aa042fa"
server_key_version: "encryptionkey@0"
I20260620 20:21:25.777822 6629 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781986536687152-17252-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781986536687152-17252-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781986536687152-17252-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:21:25.801432 6629 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:21:25.805886 6629 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:21:25.806191 6629 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:21:25.825662 6629 rpc_server.cc:307] RPC server started. Bound to: 127.16.217.62:44297
I20260620 20:21:25.825681 6695 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.217.62:44297 every 8 connection(s)
I20260620 20:21:25.827052 6629 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781986536687152-17252-0/minicluster-data/master-0/data/info.pb
I20260620 20:21:25.830714 6696 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:21:25.833599 17252 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu as pid 6629
I20260620 20:21:25.833781 17252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781986536687152-17252-0/minicluster-data/master-0/wal/instance
I20260620 20:21:25.834335 17252 external_mini_cluster.cc:1428] Setting key 6920024a3b375d3950cedde9b495d59f
I20260620 20:21:25.844336 6696 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5ce943b945594c419c213b27fb3c3b9b: Bootstrap starting.
I20260620 20:21:25.848183 6696 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5ce943b945594c419c213b27fb3c3b9b: Neither blocks nor log segments found. Creating new log.
I20260620 20:21:25.849510 6696 log.cc:824] T 00000000000000000000000000000000 P 5ce943b945594c419c213b27fb3c3b9b: Log is configured to *not* fsync() on all Append() calls
I20260620 20:21:25.849928 6699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:25.836429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46787 (local address 127.16.217.62:44297)
0620 20:21:25.837333 (+ 904us) server_negotiation.cc:206] Beginning negotiation
0620 20:21:25.837347 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 20:21:25.837757 (+ 410us) server_negotiation.cc:407] Connection header received
0620 20:21:25.839826 (+ 2069us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:21:25.839855 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:21:25.840389 (+ 534us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:21:25.840838 (+ 449us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:21:25.841382 (+ 544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:25.843397 (+ 2015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:25.844735 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:25.845747 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:25.845948 (+ 201us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:21:25.847133 (+ 1185us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:21:25.847144 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:21:25.847160 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:21:25.847872 (+ 712us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:21:25.847964 (+ 92us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:21:25.848000 (+ 36us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:21:25.848173 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0620 20:21:25.848410 (+ 237us) server_negotiation.cc:299] Negotiation successful
0620 20:21:25.848604 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":651,"thread_start_us":399,"threads_started":1}
I20260620 20:21:25.852545 6696 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5ce943b945594c419c213b27fb3c3b9b: No bootstrap required, opened a new log
I20260620 20:21:25.853665 17252 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 20:21:25.861126 6696 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5ce943b945594c419c213b27fb3c3b9b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ce943b945594c419c213b27fb3c3b9b" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 44297 } }
I20260620 20:21:25.861586 6696 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5ce943b945594c419c213b27fb3c3b9b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:21:25.861718 6696 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5ce943b945594c419c213b27fb3c3b9b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ce943b945594c419c213b27fb3c3b9b, State: Initialized, Role: FOLLOWER
I20260620 20:21:25.862486 6696 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5ce943b945594c419c213b27fb3c3b9b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ce943b945594c419c213b27fb3c3b9b" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 44297 } }
I20260620 20:21:25.862814 6696 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5ce943b945594c419c213b27fb3c3b9b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 20:21:25.863004 6696 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5ce943b945594c419c213b27fb3c3b9b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 20:21:25.863206 6696 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5ce943b945594c419c213b27fb3c3b9b [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:21:25.866605 6696 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5ce943b945594c419c213b27fb3c3b9b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ce943b945594c419c213b27fb3c3b9b" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 44297 } }
I20260620 20:21:25.867242 6696 leader_election.cc:302] T 00000000000000000000000000000000 P 5ce943b945594c419c213b27fb3c3b9b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5ce943b945594c419c213b27fb3c3b9b; no voters:
I20260620 20:21:25.867996 6696 leader_election.cc:288] T 00000000000000000000000000000000 P 5ce943b945594c419c213b27fb3c3b9b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 20:21:25.868134 6706 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5ce943b945594c419c213b27fb3c3b9b [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:21:25.869266 6699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:25.858212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33455 (local address 127.16.217.62:44297)
0620 20:21:25.858404 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0620 20:21:25.858411 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:21:25.862196 (+ 3785us) server_negotiation.cc:407] Connection header received
0620 20:21:25.862418 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:21:25.862422 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:21:25.862485 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:21:25.862593 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:21:25.863343 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:25.864137 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:25.866199 (+ 2062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:25.866656 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:25.866752 (+ 96us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:21:25.868112 (+ 1360us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:21:25.868122 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:21:25.868130 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:21:25.868153 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:21:25.868225 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:21:25.868248 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:21:25.868404 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0620 20:21:25.868718 (+ 314us) server_negotiation.cc:299] Negotiation successful
0620 20:21:25.868791 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 20:21:25.870196 6706 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5ce943b945594c419c213b27fb3c3b9b [term 1 LEADER]: Becoming Leader. State: Replica: 5ce943b945594c419c213b27fb3c3b9b, State: Running, Role: LEADER
I20260620 20:21:25.870755 6706 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5ce943b945594c419c213b27fb3c3b9b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ce943b945594c419c213b27fb3c3b9b" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 44297 } }
I20260620 20:21:25.870966 6696 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5ce943b945594c419c213b27fb3c3b9b [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:21:25.874173 6707 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5ce943b945594c419c213b27fb3c3b9b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5ce943b945594c419c213b27fb3c3b9b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ce943b945594c419c213b27fb3c3b9b" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 44297 } } }
I20260620 20:21:25.874410 6708 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5ce943b945594c419c213b27fb3c3b9b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5ce943b945594c419c213b27fb3c3b9b. Latest consensus state: current_term: 1 leader_uuid: "5ce943b945594c419c213b27fb3c3b9b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ce943b945594c419c213b27fb3c3b9b" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 44297 } } }
I20260620 20:21:25.874679 6707 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5ce943b945594c419c213b27fb3c3b9b [sys.catalog]: This master's current role is: LEADER
I20260620 20:21:25.874703 6708 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5ce943b945594c419c213b27fb3c3b9b [sys.catalog]: This master's current role is: LEADER
I20260620 20:21:25.877353 6715 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 20:21:25.883163 6715 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 20:21:25.891911 6715 catalog_manager.cc:1347] Generated new cluster ID: b8a6d3d986eb40269b201ee573382e0d
I20260620 20:21:25.892022 6715 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 20:21:25.912420 6715 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 20:21:25.914348 6715 catalog_manager.cc:1504] Loading token signing keys...
I20260620 20:21:25.924397 6715 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5ce943b945594c419c213b27fb3c3b9b: Generated new TSK 0
I20260620 20:21:25.925475 6715 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 20:21:25.939890 17252 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu pid 6629
W20260620 20:21:25.941111 6691 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:46787
W20260620 20:21:25.942044 17252 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu pid 6629: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 20:21:25.942222 17252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu with pid 6629
W20260620 20:21:25.959196 6614 connection.cc:535] client connection to 127.16.217.62:44297 recv error: Network error: failed to read from TLS socket (remote: 127.16.217.62:44297): Connection reset by peer (error 104)
2026-06-20T20:21:25Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1969 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781986536687152-17252-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 20:21:26 dist-test-slave-2hpg krb5kdc[6727](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 20:21:26 dist-test-slave-2hpg krb5kdc[6727](info): set up 2 sockets
Jun 20 20:21:26 dist-test-slave-2hpg krb5kdc[6727](info): commencing operation
krb5kdc: starting...
W20260620 20:21:27.074770 17252 mini_kdc.cc:121] Time spent starting KDC: real 1.097s user 0.003s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781986536687152-17252-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781986536687152-17252-0/krb5kdc/test-user.keytab.
Jun 20 20:21:27 dist-test-slave-2hpg krb5kdc[6727](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986887, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T20:21:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T20:21:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.217.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.217.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.217.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.62.keytab.
Entry for principal kudu/127.16.217.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.62.keytab.
I20260620 20:21:27.337823 17252 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
/tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781986536687152-17252-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781986536687152-17252-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781986536687152-17252-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781986536687152-17252-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.217.62:45867
--webserver_interface=127.16.217.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.62.keytab
--principal=kudu/127.16.217.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.217.20:42755
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.217.62:45867
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781986536687152-17252-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781986536687152-17252-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781986536687152-17252-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:21:27.673383 6740 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:21:27.673681 6740 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:21:27.673781 6740 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:21:27.681113 6740 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:21:27.681248 6740 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:21:27.681311 6740 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:21:27.681380 6740 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:21:27.681432 6740 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:21:27.691627 6740 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.217.20:42755
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781986536687152-17252-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781986536687152-17252-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.217.62:45867
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.62.keytab
--principal=kudu/127.16.217.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.217.62:45867
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781986536687152-17252-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.217.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781986536687152-17252-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:21:27.692840 6740 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:21:27.694135 6740 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:21:27.705782 6745 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:21:27.705734 6746 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:21:27.705693 6748 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:21:27.706955 6740 server_base.cc:1034] running on GCE node
I20260620 20:21:27.707724 6740 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:21:27.709071 6740 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:21:27.710301 6740 hybrid_clock.cc:648] HybridClock initialized: now 1781986887710260 us; error 31 us; skew 500 ppm
I20260620 20:21:27.710664 6740 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:21:27 dist-test-slave-2hpg krb5kdc[6727](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986887, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.217.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:21:27.713725 6740 init.cc:377] Logged in from keytab as kudu/127.16.217.62@KRBTEST.COM (short username kudu)
I20260620 20:21:27.715269 6740 webserver.cc:460] Webserver started at http://127.16.217.62:34035/ using document root <none> and password file <none>
I20260620 20:21:27.715932 6740 fs_manager.cc:359] Metadata directory not provided
I20260620 20:21:27.716078 6740 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:21:27.716425 6740 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:21:27.719377 6740 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781986536687152-17252-0/minicluster-data/master-0/data/instance:
uuid: "e95303f6424240849f408fef39b77b40"
format_stamp: "Formatted at 2026-06-20 20:21:27 on dist-test-slave-2hpg"
server_key: "28dce7d181dfa2732c0cccee6065265a"
server_key_iv: "c1b5422dbc29e6dc8317d6a3a1dfd2e2"
server_key_version: "encryptionkey@0"
I20260620 20:21:27.720161 6740 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781986536687152-17252-0/minicluster-data/master-0/wal/instance:
uuid: "e95303f6424240849f408fef39b77b40"
format_stamp: "Formatted at 2026-06-20 20:21:27 on dist-test-slave-2hpg"
server_key: "28dce7d181dfa2732c0cccee6065265a"
server_key_iv: "c1b5422dbc29e6dc8317d6a3a1dfd2e2"
server_key_version: "encryptionkey@0"
I20260620 20:21:27.725154 6740 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.006s
I20260620 20:21:27.728549 6755 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:21:27.729802 6740 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 20:21:27.730027 6740 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781986536687152-17252-0/minicluster-data/master-0/data,/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781986536687152-17252-0/minicluster-data/master-0/wal
uuid: "e95303f6424240849f408fef39b77b40"
format_stamp: "Formatted at 2026-06-20 20:21:27 on dist-test-slave-2hpg"
server_key: "28dce7d181dfa2732c0cccee6065265a"
server_key_iv: "c1b5422dbc29e6dc8317d6a3a1dfd2e2"
server_key_version: "encryptionkey@0"
I20260620 20:21:27.730249 6740 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781986536687152-17252-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781986536687152-17252-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781986536687152-17252-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:21:27.735773 6740 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:21:27.739725 6740 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:21:27.740013 6740 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:21:27.758414 6740 rpc_server.cc:307] RPC server started. Bound to: 127.16.217.62:45867
I20260620 20:21:27.758450 6806 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.217.62:45867 every 8 connection(s)
I20260620 20:21:27.759761 6740 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781986536687152-17252-0/minicluster-data/master-0/data/info.pb
I20260620 20:21:27.760843 17252 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu as pid 6740
I20260620 20:21:27.761044 17252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781986536687152-17252-0/minicluster-data/master-0/wal/instance
I20260620 20:21:27.761579 17252 external_mini_cluster.cc:1428] Setting key 02f6cdfbabf588590626e6c44a4f0c70
I20260620 20:21:27.764360 6807 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:21:27.769158 6810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:27.763370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54359 (local address 127.16.217.62:45867)
0620 20:21:27.764107 (+ 737us) server_negotiation.cc:206] Beginning negotiation
0620 20:21:27.764120 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 20:21:27.764159 (+ 39us) server_negotiation.cc:407] Connection header received
0620 20:21:27.765827 (+ 1668us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:21:27.765849 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:21:27.766221 (+ 372us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:21:27.766610 (+ 389us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:21:27.767624 (+ 1014us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:21:27.767630 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:21:27.767642 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:21:27.767655 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:21:27.767728 (+ 73us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:21:27.767753 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:21:27.767981 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0620 20:21:27.768114 (+ 133us) server_negotiation.cc:299] Negotiation successful
0620 20:21:27.768284 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":528,"thread_start_us":328,"threads_started":1}
I20260620 20:21:27.772672 17252 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 20:21:27.775489 6807 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e95303f6424240849f408fef39b77b40: Bootstrap starting.
I20260620 20:21:27.778968 6807 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e95303f6424240849f408fef39b77b40: Neither blocks nor log segments found. Creating new log.
I20260620 20:21:27.779356 6810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:27.776997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50191 (local address 127.16.217.62:45867)
0620 20:21:27.777204 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0620 20:21:27.777210 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:21:27.777531 (+ 321us) server_negotiation.cc:407] Connection header received
0620 20:21:27.777680 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:21:27.777683 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:21:27.777724 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:21:27.777844 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:21:27.778743 (+ 899us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:21:27.778748 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:21:27.778753 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:21:27.778773 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:21:27.778808 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:21:27.778823 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:21:27.778940 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0620 20:21:27.779032 (+ 92us) server_negotiation.cc:299] Negotiation successful
0620 20:21:27.779086 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 20:21:27.780221 6807 log.cc:824] T 00000000000000000000000000000000 P e95303f6424240849f408fef39b77b40: Log is configured to *not* fsync() on all Append() calls
I20260620 20:21:27.783324 6807 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e95303f6424240849f408fef39b77b40: No bootstrap required, opened a new log
I20260620 20:21:27.790951 6807 raft_consensus.cc:348] T 00000000000000000000000000000000 P e95303f6424240849f408fef39b77b40 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e95303f6424240849f408fef39b77b40" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 45867 } }
I20260620 20:21:27.791383 6807 raft_consensus.cc:374] T 00000000000000000000000000000000 P e95303f6424240849f408fef39b77b40 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:21:27.791500 6807 raft_consensus.cc:729] T 00000000000000000000000000000000 P e95303f6424240849f408fef39b77b40 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e95303f6424240849f408fef39b77b40, State: Initialized, Role: FOLLOWER
I20260620 20:21:27.792114 6807 consensus_queue.cc:260] T 00000000000000000000000000000000 P e95303f6424240849f408fef39b77b40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e95303f6424240849f408fef39b77b40" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 45867 } }
I20260620 20:21:27.792342 6807 raft_consensus.cc:388] T 00000000000000000000000000000000 P e95303f6424240849f408fef39b77b40 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 20:21:27.792500 6807 raft_consensus.cc:482] T 00000000000000000000000000000000 P e95303f6424240849f408fef39b77b40 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 20:21:27.792683 6807 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e95303f6424240849f408fef39b77b40 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:21:27.794853 6807 raft_consensus.cc:504] T 00000000000000000000000000000000 P e95303f6424240849f408fef39b77b40 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e95303f6424240849f408fef39b77b40" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 45867 } }
I20260620 20:21:27.795324 6807 leader_election.cc:302] T 00000000000000000000000000000000 P e95303f6424240849f408fef39b77b40 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e95303f6424240849f408fef39b77b40; no voters:
I20260620 20:21:27.795874 6807 leader_election.cc:288] T 00000000000000000000000000000000 P e95303f6424240849f408fef39b77b40 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 20:21:27.796089 6817 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e95303f6424240849f408fef39b77b40 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:21:27.796936 6817 raft_consensus.cc:686] T 00000000000000000000000000000000 P e95303f6424240849f408fef39b77b40 [term 1 LEADER]: Becoming Leader. State: Replica: e95303f6424240849f408fef39b77b40, State: Running, Role: LEADER
I20260620 20:21:27.797607 6817 consensus_queue.cc:237] T 00000000000000000000000000000000 P e95303f6424240849f408fef39b77b40 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e95303f6424240849f408fef39b77b40" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 45867 } }
I20260620 20:21:27.798172 6807 sys_catalog.cc:564] T 00000000000000000000000000000000 P e95303f6424240849f408fef39b77b40 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:21:27.800851 6818 sys_catalog.cc:455] T 00000000000000000000000000000000 P e95303f6424240849f408fef39b77b40 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e95303f6424240849f408fef39b77b40" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e95303f6424240849f408fef39b77b40" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 45867 } } }
I20260620 20:21:27.801353 6818 sys_catalog.cc:458] T 00000000000000000000000000000000 P e95303f6424240849f408fef39b77b40 [sys.catalog]: This master's current role is: LEADER
I20260620 20:21:27.801249 6819 sys_catalog.cc:455] T 00000000000000000000000000000000 P e95303f6424240849f408fef39b77b40 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e95303f6424240849f408fef39b77b40. Latest consensus state: current_term: 1 leader_uuid: "e95303f6424240849f408fef39b77b40" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e95303f6424240849f408fef39b77b40" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 45867 } } }
I20260620 20:21:27.801592 6819 sys_catalog.cc:458] T 00000000000000000000000000000000 P e95303f6424240849f408fef39b77b40 [sys.catalog]: This master's current role is: LEADER
I20260620 20:21:27.803369 6826 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 20:21:27.808851 6826 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 20:21:27.818638 6826 catalog_manager.cc:1347] Generated new cluster ID: 05af76b1c15a4b9cafd74c5c6cf68ba9
I20260620 20:21:27.818768 6826 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 20:21:27.830940 6826 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 20:21:27.831339 6826 catalog_manager.cc:1504] Loading token signing keys...
I20260620 20:21:27.843868 6826 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e95303f6424240849f408fef39b77b40: Generated new TSK 0
I20260620 20:21:27.844630 6826 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 20:21:27.853844 17252 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu pid 6740
W20260620 20:21:27.854882 6802 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:54359
W20260620 20:21:27.855500 17252 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu pid 6740: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 20:21:27.855607 17252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu with pid 6740
2026-06-20T20:21:27Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1913 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781986536687152-17252-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 20:21:27 dist-test-slave-2hpg krb5kdc[6838](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 20:21:27 dist-test-slave-2hpg krb5kdc[6838](info): set up 2 sockets
Jun 20 20:21:27 dist-test-slave-2hpg krb5kdc[6838](info): commencing operation
krb5kdc: starting...
W20260620 20:21:28.962954 17252 mini_kdc.cc:121] Time spent starting KDC: real 1.071s user 0.000s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781986536687152-17252-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781986536687152-17252-0/krb5kdc/test-user.keytab.
Jun 20 20:21:29 dist-test-slave-2hpg krb5kdc[6838](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986889, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T20:21:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T20:21:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.217.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.217.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.217.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.62.keytab.
Entry for principal kudu/127.16.217.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.62.keytab.
I20260620 20:21:29.233619 17252 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
/tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781986536687152-17252-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781986536687152-17252-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781986536687152-17252-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781986536687152-17252-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.217.62:35653
--webserver_interface=127.16.217.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.62.keytab
--principal=kudu/127.16.217.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.217.20:41063
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.217.62:35653
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781986536687152-17252-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781986536687152-17252-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781986536687152-17252-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:21:29.577019 6851 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:21:29.577325 6851 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:21:29.577427 6851 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:21:29.585466 6851 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:21:29.585583 6851 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:21:29.585630 6851 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:21:29.585693 6851 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:21:29.585738 6851 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:21:29.595608 6851 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.217.20:41063
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781986536687152-17252-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781986536687152-17252-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.217.62:35653
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.62.keytab
--principal=kudu/127.16.217.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.217.62:35653
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781986536687152-17252-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.217.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781986536687152-17252-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:21:29.596907 6851 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:21:29.598207 6851 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:21:29.610167 6859 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:21:29.610296 6856 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:21:29.610261 6857 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:21:29.611320 6851 server_base.cc:1034] running on GCE node
I20260620 20:21:29.612089 6851 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:21:29.613520 6851 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:21:29.614748 6851 hybrid_clock.cc:648] HybridClock initialized: now 1781986889614704 us; error 59 us; skew 500 ppm
I20260620 20:21:29.615152 6851 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:21:29 dist-test-slave-2hpg krb5kdc[6838](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986889, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.217.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:21:29.618415 6851 init.cc:377] Logged in from keytab as kudu/127.16.217.62@KRBTEST.COM (short username kudu)
I20260620 20:21:29.620000 6851 webserver.cc:460] Webserver started at http://127.16.217.62:39543/ using document root <none> and password file <none>
I20260620 20:21:29.620723 6851 fs_manager.cc:359] Metadata directory not provided
I20260620 20:21:29.620870 6851 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:21:29.621193 6851 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:21:29.624238 6851 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781986536687152-17252-0/minicluster-data/master-0/data/instance:
uuid: "c5ff9740b1c8422295531abe29cd6f5a"
format_stamp: "Formatted at 2026-06-20 20:21:29 on dist-test-slave-2hpg"
server_key: "6a81d1dee6a6df52c5b5cb377ecc567e"
server_key_iv: "1db854669be6bd48ef2264ad9e98d15a"
server_key_version: "encryptionkey@0"
I20260620 20:21:29.625007 6851 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781986536687152-17252-0/minicluster-data/master-0/wal/instance:
uuid: "c5ff9740b1c8422295531abe29cd6f5a"
format_stamp: "Formatted at 2026-06-20 20:21:29 on dist-test-slave-2hpg"
server_key: "6a81d1dee6a6df52c5b5cb377ecc567e"
server_key_iv: "1db854669be6bd48ef2264ad9e98d15a"
server_key_version: "encryptionkey@0"
I20260620 20:21:29.630383 6851 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260620 20:21:29.633924 6866 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:21:29.635331 6851 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 20:21:29.635553 6851 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781986536687152-17252-0/minicluster-data/master-0/data,/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781986536687152-17252-0/minicluster-data/master-0/wal
uuid: "c5ff9740b1c8422295531abe29cd6f5a"
format_stamp: "Formatted at 2026-06-20 20:21:29 on dist-test-slave-2hpg"
server_key: "6a81d1dee6a6df52c5b5cb377ecc567e"
server_key_iv: "1db854669be6bd48ef2264ad9e98d15a"
server_key_version: "encryptionkey@0"
I20260620 20:21:29.635756 6851 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781986536687152-17252-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781986536687152-17252-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781986536687152-17252-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:21:29.655387 6851 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:21:29.659734 6851 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:21:29.660034 6851 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:21:29.679292 6851 rpc_server.cc:307] RPC server started. Bound to: 127.16.217.62:35653
I20260620 20:21:29.679311 6917 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.217.62:35653 every 8 connection(s)
I20260620 20:21:29.680716 6851 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781986536687152-17252-0/minicluster-data/master-0/data/info.pb
I20260620 20:21:29.684444 6918 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:21:29.689173 17252 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu as pid 6851
I20260620 20:21:29.689360 17252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781986536687152-17252-0/minicluster-data/master-0/wal/instance
I20260620 20:21:29.689805 17252 external_mini_cluster.cc:1428] Setting key 40abfbf4cc8cf578ef9fe11d54e67c54
I20260620 20:21:29.695557 6918 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c5ff9740b1c8422295531abe29cd6f5a: Bootstrap starting.
I20260620 20:21:29.698700 6918 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c5ff9740b1c8422295531abe29cd6f5a: Neither blocks nor log segments found. Creating new log.
I20260620 20:21:29.699723 6918 log.cc:824] T 00000000000000000000000000000000 P c5ff9740b1c8422295531abe29cd6f5a: Log is configured to *not* fsync() on all Append() calls
I20260620 20:21:29.702705 6918 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c5ff9740b1c8422295531abe29cd6f5a: No bootstrap required, opened a new log
Jun 20 20:21:29 dist-test-slave-2hpg krb5kdc[6838](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986889, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.217.62@KRBTEST.COM
I20260620 20:21:29.713833 6921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:29.691696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34557 (local address 127.16.217.62:35653)
0620 20:21:29.692638 (+ 942us) server_negotiation.cc:206] Beginning negotiation
0620 20:21:29.692650 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 20:21:29.692694 (+ 44us) server_negotiation.cc:407] Connection header received
0620 20:21:29.694510 (+ 1816us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:21:29.694533 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:21:29.694913 (+ 380us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:21:29.695283 (+ 370us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:21:29.696736 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:29.698131 (+ 1395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:29.699432 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:29.700305 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:29.700530 (+ 225us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:21:29.705046 (+ 4516us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:21:29.705061 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:21:29.705080 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:21:29.705130 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:21:29.709291 (+ 4161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:21:29.710024 (+ 733us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:21:29.710033 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:21:29.710041 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:21:29.710260 (+ 219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:21:29.710608 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:21:29.710613 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:21:29.710616 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:21:29.711389 (+ 773us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:21:29.711692 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0620 20:21:29.712040 (+ 348us) server_negotiation.cc:299] Negotiation successful
0620 20:21:29.712483 (+ 443us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":612,"thread_start_us":384,"threads_started":1}
I20260620 20:21:29.713662 6918 raft_consensus.cc:348] T 00000000000000000000000000000000 P c5ff9740b1c8422295531abe29cd6f5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5ff9740b1c8422295531abe29cd6f5a" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 35653 } }
I20260620 20:21:29.714205 6918 raft_consensus.cc:374] T 00000000000000000000000000000000 P c5ff9740b1c8422295531abe29cd6f5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:21:29.714319 6918 raft_consensus.cc:729] T 00000000000000000000000000000000 P c5ff9740b1c8422295531abe29cd6f5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5ff9740b1c8422295531abe29cd6f5a, State: Initialized, Role: FOLLOWER
I20260620 20:21:29.714996 6918 consensus_queue.cc:260] T 00000000000000000000000000000000 P c5ff9740b1c8422295531abe29cd6f5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5ff9740b1c8422295531abe29cd6f5a" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 35653 } }
I20260620 20:21:29.715281 6918 raft_consensus.cc:388] T 00000000000000000000000000000000 P c5ff9740b1c8422295531abe29cd6f5a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 20:21:29.715404 6918 raft_consensus.cc:482] T 00000000000000000000000000000000 P c5ff9740b1c8422295531abe29cd6f5a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 20:21:29.715555 6918 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c5ff9740b1c8422295531abe29cd6f5a [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:21:29.717785 17252 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 20:21:29.718242 6918 raft_consensus.cc:504] T 00000000000000000000000000000000 P c5ff9740b1c8422295531abe29cd6f5a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5ff9740b1c8422295531abe29cd6f5a" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 35653 } }
I20260620 20:21:29.718721 6918 leader_election.cc:302] T 00000000000000000000000000000000 P c5ff9740b1c8422295531abe29cd6f5a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c5ff9740b1c8422295531abe29cd6f5a; no voters:
I20260620 20:21:29.719300 6918 leader_election.cc:288] T 00000000000000000000000000000000 P c5ff9740b1c8422295531abe29cd6f5a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 20:21:29.719461 6923 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c5ff9740b1c8422295531abe29cd6f5a [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:21:29.721251 6918 sys_catalog.cc:564] T 00000000000000000000000000000000 P c5ff9740b1c8422295531abe29cd6f5a [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:21:29.723681 6923 raft_consensus.cc:686] T 00000000000000000000000000000000 P c5ff9740b1c8422295531abe29cd6f5a [term 1 LEADER]: Becoming Leader. State: Replica: c5ff9740b1c8422295531abe29cd6f5a, State: Running, Role: LEADER
I20260620 20:21:29.725062 6923 consensus_queue.cc:237] T 00000000000000000000000000000000 P c5ff9740b1c8422295531abe29cd6f5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5ff9740b1c8422295531abe29cd6f5a" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 35653 } }
I20260620 20:21:29.728073 6925 sys_catalog.cc:455] T 00000000000000000000000000000000 P c5ff9740b1c8422295531abe29cd6f5a [sys.catalog]: SysCatalogTable state changed. Reason: New leader c5ff9740b1c8422295531abe29cd6f5a. Latest consensus state: current_term: 1 leader_uuid: "c5ff9740b1c8422295531abe29cd6f5a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5ff9740b1c8422295531abe29cd6f5a" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 35653 } } }
I20260620 20:21:29.728495 6925 sys_catalog.cc:458] T 00000000000000000000000000000000 P c5ff9740b1c8422295531abe29cd6f5a [sys.catalog]: This master's current role is: LEADER
I20260620 20:21:29.729071 6924 sys_catalog.cc:455] T 00000000000000000000000000000000 P c5ff9740b1c8422295531abe29cd6f5a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c5ff9740b1c8422295531abe29cd6f5a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5ff9740b1c8422295531abe29cd6f5a" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 35653 } } }
I20260620 20:21:29.729363 6924 sys_catalog.cc:458] T 00000000000000000000000000000000 P c5ff9740b1c8422295531abe29cd6f5a [sys.catalog]: This master's current role is: LEADER
I20260620 20:21:29.732262 6936 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 20:21:29.738698 6936 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 20:21:29.744074 6921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:29.724570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32789 (local address 127.16.217.62:35653)
0620 20:21:29.724786 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0620 20:21:29.724794 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 20:21:29.725217 (+ 423us) server_negotiation.cc:407] Connection header received
0620 20:21:29.725447 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:21:29.725458 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:21:29.725565 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:21:29.725642 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:21:29.728013 (+ 2371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:29.729367 (+ 1354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:29.731663 (+ 2296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:29.732526 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:29.732664 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:21:29.736255 (+ 3591us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:21:29.736261 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:21:29.736267 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:21:29.736304 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:21:29.740337 (+ 4033us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:21:29.741274 (+ 937us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:21:29.741279 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:21:29.741283 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:21:29.741444 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:21:29.741991 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:21:29.742013 (+ 22us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:21:29.742019 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:21:29.742510 (+ 491us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:21:29.742761 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0620 20:21:29.743431 (+ 670us) server_negotiation.cc:299] Negotiation successful
0620 20:21:29.743675 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 20:21:29.750200 6936 catalog_manager.cc:1347] Generated new cluster ID: ce854f89767240268a456be11d026b0b
I20260620 20:21:29.750330 6936 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 20:21:29.762375 6936 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 20:21:29.764273 6936 catalog_manager.cc:1504] Loading token signing keys...
I20260620 20:21:29.780599 6936 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c5ff9740b1c8422295531abe29cd6f5a: Generated new TSK 0
I20260620 20:21:29.781481 6936 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 20:21:29.793133 17252 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu pid 6851
I20260620 20:21:29.794297 6913 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:21:29.884151 17252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu with pid 6851
2026-06-20T20:21:29Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (2028 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781986536687152-17252-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 20:21:29 dist-test-slave-2hpg krb5kdc[6950](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 20:21:29 dist-test-slave-2hpg krb5kdc[6950](info): set up 2 sockets
Jun 20 20:21:29 dist-test-slave-2hpg krb5kdc[6950](info): commencing operation
krb5kdc: starting...
W20260620 20:21:30.986716 17252 mini_kdc.cc:121] Time spent starting KDC: real 1.066s user 0.000s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781986536687152-17252-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781986536687152-17252-0/krb5kdc/test-user.keytab.
Jun 20 20:21:31 dist-test-slave-2hpg krb5kdc[6950](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986891, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T20:21:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T20:21:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.217.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.217.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.217.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.62.keytab.
Entry for principal kudu/127.16.217.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.62.keytab.
I20260620 20:21:31.263414 17252 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
/tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781986536687152-17252-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781986536687152-17252-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781986536687152-17252-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781986536687152-17252-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.217.62:36435
--webserver_interface=127.16.217.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.62.keytab
--principal=kudu/127.16.217.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.217.20:37363
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.217.62:36435
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781986536687152-17252-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781986536687152-17252-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781986536687152-17252-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:21:31.598824 6963 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:21:31.599113 6963 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:21:31.599274 6963 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:21:31.606626 6963 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:21:31.606746 6963 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:21:31.606827 6963 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:21:31.606940 6963 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:21:31.607002 6963 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:21:31.617558 6963 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.217.20:37363
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781986536687152-17252-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781986536687152-17252-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.217.62:36435
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.62.keytab
--principal=kudu/127.16.217.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.217.62:36435
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781986536687152-17252-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.217.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781986536687152-17252-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:21:31.618932 6963 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:21:31.620286 6963 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:21:31.632949 6968 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:21:31.632915 6969 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:21:31.632998 6971 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:21:31.634477 6963 server_base.cc:1034] running on GCE node
I20260620 20:21:31.635371 6963 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:21:31.636857 6963 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:21:31.638069 6963 hybrid_clock.cc:648] HybridClock initialized: now 1781986891638013 us; error 38 us; skew 500 ppm
I20260620 20:21:31.638407 6963 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:21:31 dist-test-slave-2hpg krb5kdc[6950](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986891, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.217.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:21:31.644788 6963 init.cc:377] Logged in from keytab as kudu/127.16.217.62@KRBTEST.COM (short username kudu)
I20260620 20:21:31.646281 6963 webserver.cc:460] Webserver started at http://127.16.217.62:35489/ using document root <none> and password file <none>
I20260620 20:21:31.646920 6963 fs_manager.cc:359] Metadata directory not provided
I20260620 20:21:31.647027 6963 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:21:31.647274 6963 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:21:31.650319 6963 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781986536687152-17252-0/minicluster-data/master-0/data/instance:
uuid: "5713255c59744b8984b9995d6570aba3"
format_stamp: "Formatted at 2026-06-20 20:21:31 on dist-test-slave-2hpg"
server_key: "4ab77bf0b30d83d41957d04b42227892"
server_key_iv: "e3cf4e89b732ebe80d0fd8cd27504d3d"
server_key_version: "encryptionkey@0"
I20260620 20:21:31.651403 6963 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781986536687152-17252-0/minicluster-data/master-0/wal/instance:
uuid: "5713255c59744b8984b9995d6570aba3"
format_stamp: "Formatted at 2026-06-20 20:21:31 on dist-test-slave-2hpg"
server_key: "4ab77bf0b30d83d41957d04b42227892"
server_key_iv: "e3cf4e89b732ebe80d0fd8cd27504d3d"
server_key_version: "encryptionkey@0"
I20260620 20:21:31.657022 6963 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260620 20:21:31.660815 6978 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:21:31.662367 6963 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 20:21:31.662516 6963 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781986536687152-17252-0/minicluster-data/master-0/data,/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781986536687152-17252-0/minicluster-data/master-0/wal
uuid: "5713255c59744b8984b9995d6570aba3"
format_stamp: "Formatted at 2026-06-20 20:21:31 on dist-test-slave-2hpg"
server_key: "4ab77bf0b30d83d41957d04b42227892"
server_key_iv: "e3cf4e89b732ebe80d0fd8cd27504d3d"
server_key_version: "encryptionkey@0"
I20260620 20:21:31.662688 6963 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781986536687152-17252-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781986536687152-17252-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781986536687152-17252-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:21:31.679960 6963 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:21:31.684154 6963 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:21:31.684396 6963 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:21:31.703018 6963 rpc_server.cc:307] RPC server started. Bound to: 127.16.217.62:36435
I20260620 20:21:31.703040 7029 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.217.62:36435 every 8 connection(s)
I20260620 20:21:31.704555 6963 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781986536687152-17252-0/minicluster-data/master-0/data/info.pb
I20260620 20:21:31.708765 7030 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:21:31.709208 17252 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu as pid 6963
I20260620 20:21:31.709393 17252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781986536687152-17252-0/minicluster-data/master-0/wal/instance
I20260620 20:21:31.709810 17252 external_mini_cluster.cc:1428] Setting key 609d51da9927a9fe337dfa61680852b8
I20260620 20:21:31.723241 7033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:31.711544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56321 (local address 127.16.217.62:36435)
0620 20:21:31.712388 (+ 844us) server_negotiation.cc:206] Beginning negotiation
0620 20:21:31.712400 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 20:21:31.712440 (+ 40us) server_negotiation.cc:407] Connection header received
0620 20:21:31.714043 (+ 1603us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:21:31.714064 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:21:31.714483 (+ 419us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:21:31.714800 (+ 317us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:21:31.715519 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:31.716843 (+ 1324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:31.718756 (+ 1913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:31.719439 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:31.719609 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:21:31.720940 (+ 1331us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:21:31.720947 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:21:31.720959 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:21:31.721539 (+ 580us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:21:31.721615 (+ 76us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:21:31.721639 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:21:31.721721 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0620 20:21:31.722019 (+ 298us) server_negotiation.cc:299] Negotiation successful
0620 20:21:31.722291 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":638,"thread_start_us":419,"threads_started":1}
I20260620 20:21:31.723409 7030 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5713255c59744b8984b9995d6570aba3: Bootstrap starting.
I20260620 20:21:31.727135 17252 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 20:21:31.727355 7030 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5713255c59744b8984b9995d6570aba3: Neither blocks nor log segments found. Creating new log.
I20260620 20:21:31.728415 7030 log.cc:824] T 00000000000000000000000000000000 P 5713255c59744b8984b9995d6570aba3: Log is configured to *not* fsync() on all Append() calls
I20260620 20:21:31.731925 7030 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5713255c59744b8984b9995d6570aba3: No bootstrap required, opened a new log
I20260620 20:21:31.737628 7033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:31.731458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54759 (local address 127.16.217.62:36435)
0620 20:21:31.731628 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0620 20:21:31.731633 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:21:31.732162 (+ 529us) server_negotiation.cc:407] Connection header received
0620 20:21:31.732384 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:21:31.732388 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:21:31.732452 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:21:31.732560 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:21:31.733127 (+ 567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:31.733914 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:31.735206 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:31.735667 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:31.735797 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:21:31.736919 (+ 1122us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:21:31.736924 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:21:31.736926 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:21:31.736942 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:21:31.736981 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:21:31.736995 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:21:31.737103 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0620 20:21:31.737245 (+ 142us) server_negotiation.cc:299] Negotiation successful
0620 20:21:31.737337 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 20:21:31.740629 7030 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5713255c59744b8984b9995d6570aba3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5713255c59744b8984b9995d6570aba3" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 36435 } }
I20260620 20:21:31.741060 7030 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5713255c59744b8984b9995d6570aba3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:21:31.741187 7030 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5713255c59744b8984b9995d6570aba3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5713255c59744b8984b9995d6570aba3, State: Initialized, Role: FOLLOWER
I20260620 20:21:31.741966 7030 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5713255c59744b8984b9995d6570aba3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5713255c59744b8984b9995d6570aba3" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 36435 } }
I20260620 20:21:31.742305 7030 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5713255c59744b8984b9995d6570aba3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 20:21:31.742503 7030 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5713255c59744b8984b9995d6570aba3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 20:21:31.742693 7030 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5713255c59744b8984b9995d6570aba3 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:21:31.745167 7030 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5713255c59744b8984b9995d6570aba3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5713255c59744b8984b9995d6570aba3" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 36435 } }
I20260620 20:21:31.745748 7030 leader_election.cc:302] T 00000000000000000000000000000000 P 5713255c59744b8984b9995d6570aba3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5713255c59744b8984b9995d6570aba3; no voters:
I20260620 20:21:31.746376 7030 leader_election.cc:288] T 00000000000000000000000000000000 P 5713255c59744b8984b9995d6570aba3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 20:21:31.746472 7040 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5713255c59744b8984b9995d6570aba3 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:21:31.747232 7040 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5713255c59744b8984b9995d6570aba3 [term 1 LEADER]: Becoming Leader. State: Replica: 5713255c59744b8984b9995d6570aba3, State: Running, Role: LEADER
I20260620 20:21:31.747797 7040 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5713255c59744b8984b9995d6570aba3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5713255c59744b8984b9995d6570aba3" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 36435 } }
I20260620 20:21:31.748589 7030 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5713255c59744b8984b9995d6570aba3 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:21:31.750741 7042 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5713255c59744b8984b9995d6570aba3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5713255c59744b8984b9995d6570aba3. Latest consensus state: current_term: 1 leader_uuid: "5713255c59744b8984b9995d6570aba3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5713255c59744b8984b9995d6570aba3" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 36435 } } }
I20260620 20:21:31.751286 7042 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5713255c59744b8984b9995d6570aba3 [sys.catalog]: This master's current role is: LEADER
I20260620 20:21:31.751259 7041 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5713255c59744b8984b9995d6570aba3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5713255c59744b8984b9995d6570aba3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5713255c59744b8984b9995d6570aba3" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 36435 } } }
I20260620 20:21:31.751559 7041 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5713255c59744b8984b9995d6570aba3 [sys.catalog]: This master's current role is: LEADER
I20260620 20:21:31.752554 7048 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 20:21:31.757813 7048 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 20:21:31.767069 7048 catalog_manager.cc:1347] Generated new cluster ID: 0d3980858a5b4cde8b61ee605b0915a4
I20260620 20:21:31.767175 7048 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 20:21:31.790272 7048 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 20:21:31.791692 7048 catalog_manager.cc:1504] Loading token signing keys...
I20260620 20:21:31.797449 7048 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5713255c59744b8984b9995d6570aba3: Generated new TSK 0
I20260620 20:21:31.798558 7048 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 20:21:31.817993 17252 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu pid 6963
W20260620 20:21:31.819108 7025 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:56321
W20260620 20:21:31.819901 17252 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu pid 6963: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 20:21:31.820075 17252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu with pid 6963
2026-06-20T20:21:31Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1932 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781986536687152-17252-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 20:21:31 dist-test-slave-2hpg krb5kdc[7061](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 20:21:31 dist-test-slave-2hpg krb5kdc[7061](info): set up 2 sockets
Jun 20 20:21:31 dist-test-slave-2hpg krb5kdc[7061](info): commencing operation
krb5kdc: starting...
W20260620 20:21:32.930693 17252 mini_kdc.cc:121] Time spent starting KDC: real 1.078s user 0.005s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781986536687152-17252-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781986536687152-17252-0/krb5kdc/test-user.keytab.
Jun 20 20:21:33 dist-test-slave-2hpg krb5kdc[7061](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986893, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T20:21:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T20:21:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.217.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.217.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.217.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.62.keytab.
Entry for principal kudu/127.16.217.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.62.keytab.
I20260620 20:21:33.193348 17252 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
/tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781986536687152-17252-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781986536687152-17252-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781986536687152-17252-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781986536687152-17252-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.217.62:35989
--webserver_interface=127.16.217.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.62.keytab
--principal=kudu/127.16.217.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.217.20:44177
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.217.62:35989
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781986536687152-17252-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781986536687152-17252-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781986536687152-17252-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:21:33.520952 7074 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:21:33.521310 7074 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:21:33.521479 7074 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:21:33.528656 7074 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:21:33.528806 7074 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:21:33.528908 7074 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:21:33.529012 7074 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:21:33.529100 7074 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:21:33.538790 7074 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.217.20:44177
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781986536687152-17252-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781986536687152-17252-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.217.62:35989
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.62.keytab
--principal=kudu/127.16.217.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.217.62:35989
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781986536687152-17252-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.217.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781986536687152-17252-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:21:33.540083 7074 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:21:33.541399 7074 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:21:33.553241 7080 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:21:33.553275 7079 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:21:33.553339 7082 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:21:33.553956 7074 server_base.cc:1034] running on GCE node
I20260620 20:21:33.554756 7074 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:21:33.556118 7074 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:21:33.557365 7074 hybrid_clock.cc:648] HybridClock initialized: now 1781986893557326 us; error 50 us; skew 500 ppm
I20260620 20:21:33.557729 7074 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:21:33 dist-test-slave-2hpg krb5kdc[7061](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986893, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.217.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:21:33.560937 7074 init.cc:377] Logged in from keytab as kudu/127.16.217.62@KRBTEST.COM (short username kudu)
I20260620 20:21:33.562464 7074 webserver.cc:460] Webserver started at http://127.16.217.62:43157/ using document root <none> and password file <none>
I20260620 20:21:33.563129 7074 fs_manager.cc:359] Metadata directory not provided
I20260620 20:21:33.563277 7074 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:21:33.563585 7074 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:21:33.566814 7074 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781986536687152-17252-0/minicluster-data/master-0/data/instance:
uuid: "3da2e96ce2ef4259b0d38f0c011fdd6d"
format_stamp: "Formatted at 2026-06-20 20:21:33 on dist-test-slave-2hpg"
server_key: "3dd91f136c7fb7a45bfd5fc20a254758"
server_key_iv: "c3f549bafa98f8a704fe820db4894198"
server_key_version: "encryptionkey@0"
I20260620 20:21:33.567729 7074 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781986536687152-17252-0/minicluster-data/master-0/wal/instance:
uuid: "3da2e96ce2ef4259b0d38f0c011fdd6d"
format_stamp: "Formatted at 2026-06-20 20:21:33 on dist-test-slave-2hpg"
server_key: "3dd91f136c7fb7a45bfd5fc20a254758"
server_key_iv: "c3f549bafa98f8a704fe820db4894198"
server_key_version: "encryptionkey@0"
I20260620 20:21:33.572780 7074 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260620 20:21:33.576270 7089 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:21:33.577663 7074 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260620 20:21:33.577911 7074 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781986536687152-17252-0/minicluster-data/master-0/data,/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781986536687152-17252-0/minicluster-data/master-0/wal
uuid: "3da2e96ce2ef4259b0d38f0c011fdd6d"
format_stamp: "Formatted at 2026-06-20 20:21:33 on dist-test-slave-2hpg"
server_key: "3dd91f136c7fb7a45bfd5fc20a254758"
server_key_iv: "c3f549bafa98f8a704fe820db4894198"
server_key_version: "encryptionkey@0"
I20260620 20:21:33.578147 7074 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781986536687152-17252-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781986536687152-17252-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781986536687152-17252-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:21:33.584306 7074 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:21:33.588999 7074 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:21:33.589320 7074 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:21:33.609237 7074 rpc_server.cc:307] RPC server started. Bound to: 127.16.217.62:35989
I20260620 20:21:33.609256 7140 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.217.62:35989 every 8 connection(s)
I20260620 20:21:33.610800 7074 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781986536687152-17252-0/minicluster-data/master-0/data/info.pb
I20260620 20:21:33.614585 7141 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:21:33.619220 17252 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu as pid 7074
I20260620 20:21:33.619419 17252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781986536687152-17252-0/minicluster-data/master-0/wal/instance
I20260620 20:21:33.619789 17252 external_mini_cluster.cc:1428] Setting key 17f3353946559d8e71d775e8200f6d72
I20260620 20:21:33.625845 7141 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3da2e96ce2ef4259b0d38f0c011fdd6d: Bootstrap starting.
I20260620 20:21:33.627813 7144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:33.621469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33295 (local address 127.16.217.62:35989)
0620 20:21:33.622321 (+ 852us) server_negotiation.cc:206] Beginning negotiation
0620 20:21:33.622331 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 20:21:33.622371 (+ 40us) server_negotiation.cc:407] Connection header received
0620 20:21:33.624431 (+ 2060us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:21:33.624462 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:21:33.624786 (+ 324us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:21:33.625108 (+ 322us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:21:33.626224 (+ 1116us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:21:33.626231 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:21:33.626243 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:21:33.626266 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:21:33.626333 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:21:33.626363 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:21:33.626577 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0620 20:21:33.626680 (+ 103us) server_negotiation.cc:299] Negotiation successful
0620 20:21:33.626883 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":645,"thread_start_us":442,"threads_started":1}
I20260620 20:21:33.629184 7141 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3da2e96ce2ef4259b0d38f0c011fdd6d: Neither blocks nor log segments found. Creating new log.
I20260620 20:21:33.630519 7141 log.cc:824] T 00000000000000000000000000000000 P 3da2e96ce2ef4259b0d38f0c011fdd6d: Log is configured to *not* fsync() on all Append() calls
I20260620 20:21:33.631726 17252 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 20:21:33.633226 7141 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3da2e96ce2ef4259b0d38f0c011fdd6d: No bootstrap required, opened a new log
I20260620 20:21:33.638638 7144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:33.636070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33455 (local address 127.16.217.62:35989)
0620 20:21:33.636269 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0620 20:21:33.636276 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:21:33.636690 (+ 414us) server_negotiation.cc:407] Connection header received
0620 20:21:33.636880 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:21:33.636887 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:21:33.636936 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:21:33.637062 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:21:33.638012 (+ 950us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:21:33.638017 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:21:33.638020 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:21:33.638035 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:21:33.638071 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:21:33.638086 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:21:33.638202 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0620 20:21:33.638309 (+ 107us) server_negotiation.cc:299] Negotiation successful
0620 20:21:33.638371 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 20:21:33.642925 7141 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3da2e96ce2ef4259b0d38f0c011fdd6d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3da2e96ce2ef4259b0d38f0c011fdd6d" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 35989 } }
I20260620 20:21:33.643422 7141 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3da2e96ce2ef4259b0d38f0c011fdd6d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:21:33.643551 7141 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3da2e96ce2ef4259b0d38f0c011fdd6d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3da2e96ce2ef4259b0d38f0c011fdd6d, State: Initialized, Role: FOLLOWER
I20260620 20:21:33.644140 7141 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3da2e96ce2ef4259b0d38f0c011fdd6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3da2e96ce2ef4259b0d38f0c011fdd6d" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 35989 } }
I20260620 20:21:33.644397 7141 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3da2e96ce2ef4259b0d38f0c011fdd6d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 20:21:33.644586 7141 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3da2e96ce2ef4259b0d38f0c011fdd6d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 20:21:33.644781 7141 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3da2e96ce2ef4259b0d38f0c011fdd6d [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:21:33.647542 7141 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3da2e96ce2ef4259b0d38f0c011fdd6d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3da2e96ce2ef4259b0d38f0c011fdd6d" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 35989 } }
I20260620 20:21:33.648073 7141 leader_election.cc:302] T 00000000000000000000000000000000 P 3da2e96ce2ef4259b0d38f0c011fdd6d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3da2e96ce2ef4259b0d38f0c011fdd6d; no voters:
I20260620 20:21:33.648662 7141 leader_election.cc:288] T 00000000000000000000000000000000 P 3da2e96ce2ef4259b0d38f0c011fdd6d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 20:21:33.648764 7151 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3da2e96ce2ef4259b0d38f0c011fdd6d [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:21:33.649554 7151 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3da2e96ce2ef4259b0d38f0c011fdd6d [term 1 LEADER]: Becoming Leader. State: Replica: 3da2e96ce2ef4259b0d38f0c011fdd6d, State: Running, Role: LEADER
I20260620 20:21:33.650122 7151 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3da2e96ce2ef4259b0d38f0c011fdd6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3da2e96ce2ef4259b0d38f0c011fdd6d" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 35989 } }
I20260620 20:21:33.651017 7141 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3da2e96ce2ef4259b0d38f0c011fdd6d [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:21:33.652760 7153 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3da2e96ce2ef4259b0d38f0c011fdd6d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3da2e96ce2ef4259b0d38f0c011fdd6d. Latest consensus state: current_term: 1 leader_uuid: "3da2e96ce2ef4259b0d38f0c011fdd6d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3da2e96ce2ef4259b0d38f0c011fdd6d" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 35989 } } }
I20260620 20:21:33.653172 7153 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3da2e96ce2ef4259b0d38f0c011fdd6d [sys.catalog]: This master's current role is: LEADER
I20260620 20:21:33.653558 7152 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3da2e96ce2ef4259b0d38f0c011fdd6d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3da2e96ce2ef4259b0d38f0c011fdd6d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3da2e96ce2ef4259b0d38f0c011fdd6d" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 35989 } } }
I20260620 20:21:33.653832 7152 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3da2e96ce2ef4259b0d38f0c011fdd6d [sys.catalog]: This master's current role is: LEADER
I20260620 20:21:33.655299 7157 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 20:21:33.660544 7157 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 20:21:33.670099 7157 catalog_manager.cc:1347] Generated new cluster ID: a2ebcb7b7e70412cb0df17e2ddeb26d6
I20260620 20:21:33.670233 7157 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 20:21:33.712924 7157 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 20:21:33.713372 7157 catalog_manager.cc:1504] Loading token signing keys...
I20260620 20:21:33.725464 7157 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3da2e96ce2ef4259b0d38f0c011fdd6d: Generated new TSK 0
I20260620 20:21:33.726321 7157 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 20:21:33.738765 17252 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu pid 7074
W20260620 20:21:33.739787 7136 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:33295
W20260620 20:21:33.740504 17252 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu pid 7074: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 20:21:33.740634 17252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu with pid 7074
2026-06-20T20:21:33Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1923 ms)
[----------] 18 tests from AuthTokenIssuingTest (35236 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986536687152-17252-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 20:21:33 dist-test-slave-2hpg krb5kdc[7172](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 20:21:33 dist-test-slave-2hpg krb5kdc[7172](info): set up 2 sockets
Jun 20 20:21:33 dist-test-slave-2hpg krb5kdc[7172](info): commencing operation
krb5kdc: starting...
W20260620 20:21:34.850826 17252 mini_kdc.cc:121] Time spent starting KDC: real 1.074s user 0.000s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986536687152-17252-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986536687152-17252-0/krb5kdc/test-user.keytab.
Jun 20 20:21:35 dist-test-slave-2hpg krb5kdc[7172](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986895, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T20:21:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T20:21:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.217.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.217.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.217.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.62.keytab.
Entry for principal kudu/127.16.217.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.62.keytab.
I20260620 20:21:35.110042 17252 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
/tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986536687152-17252-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986536687152-17252-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986536687152-17252-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986536687152-17252-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.217.62:42485
--webserver_interface=127.16.217.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.62.keytab
--principal=kudu/127.16.217.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.217.20:45927
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.217.62:42485,127.16.217.61:36701,127.16.217.60:39253
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986536687152-17252-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986536687152-17252-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986536687152-17252-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:21:35.442739 7185 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 20:21:35.443096 7185 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:21:35.443219 7185 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:21:35.443354 7185 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:21:35.450587 7185 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:21:35.450699 7185 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:21:35.450820 7185 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:21:35.450904 7185 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:21:35.450992 7185 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:21:35.460834 7185 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.217.20:45927
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986536687152-17252-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986536687152-17252-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.217.62:42485,127.16.217.61:36701,127.16.217.60:39253
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.62.keytab
--principal=kudu/127.16.217.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.217.62:42485
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986536687152-17252-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.217.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986536687152-17252-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:21:35.462191 7185 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:21:35.463600 7185 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:21:35.475405 7190 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:21:35.475330 7193 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:21:35.475375 7191 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:21:35.476737 7185 server_base.cc:1034] running on GCE node
I20260620 20:21:35.477605 7185 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:21:35.479063 7185 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:21:35.480334 7185 hybrid_clock.cc:648] HybridClock initialized: now 1781986895480275 us; error 37 us; skew 500 ppm
I20260620 20:21:35.480834 7185 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:21:35 dist-test-slave-2hpg krb5kdc[7172](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986895, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.217.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:21:35.484215 7185 init.cc:377] Logged in from keytab as kudu/127.16.217.62@KRBTEST.COM (short username kudu)
I20260620 20:21:35.485800 7185 webserver.cc:460] Webserver started at http://127.16.217.62:44339/ using document root <none> and password file <none>
I20260620 20:21:35.486545 7185 fs_manager.cc:359] Metadata directory not provided
I20260620 20:21:35.486673 7185 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:21:35.486990 7185 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:21:35.490002 7185 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986536687152-17252-0/minicluster-data/master-0/data/instance:
uuid: "f180af9b6d7043d7a7ba302ffedeeb3a"
format_stamp: "Formatted at 2026-06-20 20:21:35 on dist-test-slave-2hpg"
server_key: "970238498fbcf8f16250dc2d6d94ed9c"
server_key_iv: "76cead626f180abccfea2191bd2f848b"
server_key_version: "encryptionkey@0"
I20260620 20:21:35.490811 7185 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986536687152-17252-0/minicluster-data/master-0/wal/instance:
uuid: "f180af9b6d7043d7a7ba302ffedeeb3a"
format_stamp: "Formatted at 2026-06-20 20:21:35 on dist-test-slave-2hpg"
server_key: "970238498fbcf8f16250dc2d6d94ed9c"
server_key_iv: "76cead626f180abccfea2191bd2f848b"
server_key_version: "encryptionkey@0"
I20260620 20:21:35.496291 7185 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260620 20:21:35.499872 7200 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:21:35.501216 7185 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260620 20:21:35.501444 7185 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986536687152-17252-0/minicluster-data/master-0/data,/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986536687152-17252-0/minicluster-data/master-0/wal
uuid: "f180af9b6d7043d7a7ba302ffedeeb3a"
format_stamp: "Formatted at 2026-06-20 20:21:35 on dist-test-slave-2hpg"
server_key: "970238498fbcf8f16250dc2d6d94ed9c"
server_key_iv: "76cead626f180abccfea2191bd2f848b"
server_key_version: "encryptionkey@0"
I20260620 20:21:35.501674 7185 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986536687152-17252-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986536687152-17252-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986536687152-17252-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:21:35.517122 7185 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:21:35.521540 7185 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:21:35.521840 7185 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:21:35.542145 7185 rpc_server.cc:307] RPC server started. Bound to: 127.16.217.62:42485
I20260620 20:21:35.542168 7251 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.217.62:42485 every 8 connection(s)
I20260620 20:21:35.543643 7185 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986536687152-17252-0/minicluster-data/master-0/data/info.pb
I20260620 20:21:35.544495 17252 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu as pid 7185
I20260620 20:21:35.544759 17252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986536687152-17252-0/minicluster-data/master-0/wal/instance
I20260620 20:21:35.545496 17252 external_mini_cluster.cc:1428] Setting key bd281263a596d2db487af60747bec7b6
I20260620 20:21:35.547851 7252 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:21:35.554723 7252 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.217.62" port: 42485 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.16.217.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.217.61@KRBTEST.COM" created.
Jun 20 20:21:35 dist-test-slave-2hpg krb5kdc[7172](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986895, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.217.62@KRBTEST.COM for kudu/127.16.217.62@KRBTEST.COM
I20260620 20:21:35.581396 7256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:35.557831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48830 (local address 127.16.217.62:42485)
0620 20:21:35.559476 (+ 1645us) server_negotiation.cc:206] Beginning negotiation
0620 20:21:35.559487 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 20:21:35.559549 (+ 62us) server_negotiation.cc:407] Connection header received
0620 20:21:35.561288 (+ 1739us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:21:35.561299 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:21:35.561608 (+ 309us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:21:35.561808 (+ 200us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:21:35.563079 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:35.565069 (+ 1990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:35.566612 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:35.567675 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:35.567880 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:21:35.574255 (+ 6375us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:21:35.574263 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:21:35.574274 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:21:35.574306 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:21:35.577111 (+ 2805us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:21:35.577808 (+ 697us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:21:35.577815 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:21:35.577817 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:21:35.578011 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:21:35.578385 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:21:35.578389 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:21:35.578393 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:21:35.578900 (+ 507us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:21:35.579090 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0620 20:21:35.579630 (+ 540us) server_negotiation.cc:299] Negotiation successful
0620 20:21:35.579940 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":1438,"spinlock_wait_cycles":22272,"thread_start_us":1177,"threads_started":1}
I20260620 20:21:35.581416 7255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:35.557831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:42485 (local address 127.0.0.1:48830)
0620 20:21:35.558895 (+ 1064us) negotiation.cc:107] Waiting for socket to connect
0620 20:21:35.558995 (+ 100us) client_negotiation.cc:174] Beginning negotiation
0620 20:21:35.560885 (+ 1890us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:21:35.561860 (+ 975us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:21:35.561912 (+ 52us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:21:35.562508 (+ 596us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:21:35.562858 (+ 350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:21:35.562874 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:21:35.565245 (+ 2371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:21:35.565249 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:21:35.566450 (+ 1201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:21:35.566466 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:21:35.567921 (+ 1455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:21:35.567926 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:21:35.568177 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:21:35.568915 (+ 738us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:21:35.568940 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:21:35.574033 (+ 5093us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:21:35.577243 (+ 3210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:21:35.577256 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:21:35.577268 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:21:35.577672 (+ 404us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:21:35.578117 (+ 445us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:21:35.578123 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:21:35.578127 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:21:35.578274 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:21:35.579045 (+ 771us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:21:35.579053 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:21:35.579248 (+ 195us) client_negotiation.cc:769] Sending connection context
0620 20:21:35.579489 (+ 241us) client_negotiation.cc:240] Negotiation successful
0620 20:21:35.580061 (+ 572us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":703,"mutex_wait_us":216,"thread_start_us":380,"threads_started":1}
I20260620 20:21:35.584575 7252 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.217.61" port: 36701 } has no permanent_uuid. Determining permanent_uuid...
I20260620 20:21:35.585597 7255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:35.585218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.61:36701 (local address 127.0.0.1:46044)
0620 20:21:35.585378 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0620 20:21:35.585477 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.217.61:36701: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260620 20:21:35.586009 7201 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.61:36701: connect: Connection refused (error 111)
I20260620 20:21:35.587632 7255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:35.587240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.61:36701 (local address 127.0.0.1:46048)
0620 20:21:35.587384 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0620 20:21:35.587498 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.217.61:36701: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260620 20:21:35.587944 7252 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.217.61:36701: Network error: Client connection negotiation failed: client connection to 127.16.217.61:36701: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.217.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.61.keytab.
Entry for principal kudu/127.16.217.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.61.keytab.
I20260620 20:21:35.605674 17252 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
/tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986536687152-17252-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986536687152-17252-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986536687152-17252-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986536687152-17252-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.217.61:36701
--webserver_interface=127.16.217.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.61.keytab
--principal=kudu/127.16.217.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.217.20:45927
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.217.62:42485,127.16.217.61:36701,127.16.217.60:39253
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986536687152-17252-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986536687152-17252-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986536687152-17252-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 20:21:35.637337 7252 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.217.61" port: 36701 } attempt: 1
I20260620 20:21:35.638772 7255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:35.638185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.61:36701 (local address 127.0.0.1:46050)
0620 20:21:35.638437 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0620 20:21:35.638580 (+ 143us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.217.61:36701: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260620 20:21:35.639844 7255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:35.639431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.61:36701 (local address 127.0.0.1:46064)
0620 20:21:35.639581 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0620 20:21:35.639670 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.217.61:36701: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260620 20:21:35.640046 7252 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.217.61:36701: Network error: Client connection negotiation failed: client connection to 127.16.217.61:36701: connect: Connection refused (error 111)
I20260620 20:21:35.708313 7252 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.217.61" port: 36701 } attempt: 2
I20260620 20:21:35.709611 7255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:35.709013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.61:36701 (local address 127.0.0.1:46072)
0620 20:21:35.709297 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0620 20:21:35.709420 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.217.61:36701: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260620 20:21:35.710865 7255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:35.710410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.61:36701 (local address 127.0.0.1:46080)
0620 20:21:35.710574 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0620 20:21:35.710687 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.217.61:36701: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260620 20:21:35.711122 7252 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.217.61:36701: Network error: Client connection negotiation failed: client connection to 127.16.217.61:36701: connect: Connection refused (error 111)
I20260620 20:21:35.802439 7252 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.217.61" port: 36701 } attempt: 3
I20260620 20:21:35.803807 7255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:35.803186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.61:36701 (local address 127.0.0.1:46088)
0620 20:21:35.803482 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0620 20:21:35.803615 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.217.61:36701: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260620 20:21:35.805238 7255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:35.804724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.61:36701 (local address 127.0.0.1:46104)
0620 20:21:35.804923 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0620 20:21:35.805041 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.217.61:36701: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260620 20:21:35.805439 7252 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.217.61:36701: Network error: Client connection negotiation failed: client connection to 127.16.217.61:36701: connect: Connection refused (error 111)
I20260620 20:21:35.948745 7252 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.217.61" port: 36701 } attempt: 4
I20260620 20:21:35.950081 7255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:35.949399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.61:36701 (local address 127.0.0.1:46106)
0620 20:21:35.949729 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0620 20:21:35.949848 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.217.61:36701: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153}
I20260620 20:21:35.951241 7255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:35.950788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.61:36701 (local address 127.0.0.1:46110)
0620 20:21:35.950952 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0620 20:21:35.951072 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.217.61:36701: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260620 20:21:35.951047 7259 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 20:21:35.951445 7252 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.217.61:36701: Network error: Client connection negotiation failed: client connection to 127.16.217.61:36701: connect: Connection refused (error 111)
W20260620 20:21:35.951514 7259 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:21:35.951632 7259 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:21:35.951731 7259 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:21:35.959313 7259 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:21:35.959405 7259 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:21:35.959519 7259 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:21:35.959604 7259 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:21:35.959733 7259 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:21:35.970108 7259 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.217.20:45927
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986536687152-17252-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986536687152-17252-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.16.217.62:42485,127.16.217.61:36701,127.16.217.60:39253
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.61.keytab
--principal=kudu/127.16.217.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.217.61:36701
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986536687152-17252-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.16.217.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986536687152-17252-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:21:35.971441 7259 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:21:35.972858 7259 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:21:35.985090 7267 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:21:35.985200 7264 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:21:35.985108 7265 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:21:35.986253 7259 server_base.cc:1034] running on GCE node
I20260620 20:21:35.987008 7259 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:21:35.988297 7259 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:21:35.989542 7259 hybrid_clock.cc:648] HybridClock initialized: now 1781986895989473 us; error 43 us; skew 500 ppm
I20260620 20:21:35.989902 7259 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:21:35 dist-test-slave-2hpg krb5kdc[7172](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986895, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.217.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:21:35.993101 7259 init.cc:377] Logged in from keytab as kudu/127.16.217.61@KRBTEST.COM (short username kudu)
I20260620 20:21:35.994558 7259 webserver.cc:460] Webserver started at http://127.16.217.61:44707/ using document root <none> and password file <none>
I20260620 20:21:35.995231 7259 fs_manager.cc:359] Metadata directory not provided
I20260620 20:21:35.995361 7259 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:21:35.995671 7259 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:21:35.998633 7259 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986536687152-17252-0/minicluster-data/master-1/data/instance:
uuid: "b8e62a119beb46cea5763e672f95303a"
format_stamp: "Formatted at 2026-06-20 20:21:35 on dist-test-slave-2hpg"
server_key: "8611b4f980755204b936080cb4980f88"
server_key_iv: "1ad847b070ac6585bef050499dcb5f61"
server_key_version: "encryptionkey@0"
I20260620 20:21:35.999450 7259 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986536687152-17252-0/minicluster-data/master-1/wal/instance:
uuid: "b8e62a119beb46cea5763e672f95303a"
format_stamp: "Formatted at 2026-06-20 20:21:35 on dist-test-slave-2hpg"
server_key: "8611b4f980755204b936080cb4980f88"
server_key_iv: "1ad847b070ac6585bef050499dcb5f61"
server_key_version: "encryptionkey@0"
I20260620 20:21:36.004714 7259 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 20:21:36.008355 7274 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:21:36.009954 7259 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 20:21:36.010186 7259 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986536687152-17252-0/minicluster-data/master-1/data,/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986536687152-17252-0/minicluster-data/master-1/wal
uuid: "b8e62a119beb46cea5763e672f95303a"
format_stamp: "Formatted at 2026-06-20 20:21:35 on dist-test-slave-2hpg"
server_key: "8611b4f980755204b936080cb4980f88"
server_key_iv: "1ad847b070ac6585bef050499dcb5f61"
server_key_version: "encryptionkey@0"
I20260620 20:21:36.010421 7259 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986536687152-17252-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986536687152-17252-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986536687152-17252-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:21:36.032390 7259 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:21:36.037194 7259 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:21:36.037487 7259 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:21:36.056452 7259 rpc_server.cc:307] RPC server started. Bound to: 127.16.217.61:36701
I20260620 20:21:36.056471 7325 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.217.61:36701 every 8 connection(s)
I20260620 20:21:36.057842 7259 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986536687152-17252-0/minicluster-data/master-1/data/info.pb
I20260620 20:21:36.060462 17252 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu as pid 7259
I20260620 20:21:36.060694 17252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986536687152-17252-0/minicluster-data/master-1/wal/instance
I20260620 20:21:36.061122 17252 external_mini_cluster.cc:1428] Setting key ac3b9ed3aa5f782e931c22269eb225a2
I20260620 20:21:36.061602 7326 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:21:36.068009 7326 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.217.62" port: 42485 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 20:21:36 dist-test-slave-2hpg krb5kdc[7172](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986895, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.217.61@KRBTEST.COM for kudu/127.16.217.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.16.217.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.217.60@KRBTEST.COM" created.
I20260620 20:21:36.087545 7256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:36.070704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48834 (local address 127.16.217.62:42485)
0620 20:21:36.070901 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0620 20:21:36.070905 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:21:36.072064 (+ 1159us) server_negotiation.cc:407] Connection header received
0620 20:21:36.073946 (+ 1882us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:21:36.073952 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:21:36.074044 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:21:36.074155 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:21:36.075594 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:36.076255 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:36.077747 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:36.078220 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:36.078356 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:21:36.081706 (+ 3350us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:21:36.081711 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:21:36.081714 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:21:36.081746 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:21:36.084447 (+ 2701us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:21:36.085189 (+ 742us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:21:36.085196 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:21:36.085199 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:21:36.085322 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:21:36.085800 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:21:36.085805 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:21:36.085809 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:21:36.086141 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:21:36.086300 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0620 20:21:36.086973 (+ 673us) server_negotiation.cc:299] Negotiation successful
0620 20:21:36.087198 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 20:21:36.088806 7329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:36.071149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:42485 (local address 127.0.0.1:48834)
0620 20:21:36.071885 (+ 736us) negotiation.cc:107] Waiting for socket to connect
0620 20:21:36.071926 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 20:21:36.073668 (+ 1742us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:21:36.074385 (+ 717us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:21:36.074393 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:21:36.074881 (+ 488us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:21:36.075420 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:21:36.075436 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:21:36.076420 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:21:36.076424 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:21:36.077587 (+ 1163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:21:36.077598 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:21:36.078445 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:21:36.078452 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:21:36.078679 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:21:36.079347 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:21:36.079369 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:21:36.081487 (+ 2118us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:21:36.084626 (+ 3139us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:21:36.084636 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:21:36.084652 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:21:36.085069 (+ 417us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:21:36.085437 (+ 368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:21:36.085441 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:21:36.085445 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:21:36.085618 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:21:36.086275 (+ 657us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:21:36.086284 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:21:36.086633 (+ 349us) client_negotiation.cc:769] Sending connection context
0620 20:21:36.086897 (+ 264us) client_negotiation.cc:240] Negotiation successful
0620 20:21:36.087409 (+ 512us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":508,"thread_start_us":319,"threads_started":1}
I20260620 20:21:36.090837 7326 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.217.61" port: 36701 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 20:21:36 dist-test-slave-2hpg krb5kdc[7172](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986895, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.217.61@KRBTEST.COM for kudu/127.16.217.61@KRBTEST.COM
I20260620 20:21:36.105605 7329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:36.091495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.61:36701 (local address 127.0.0.1:46122)
0620 20:21:36.091685 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0620 20:21:36.091705 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 20:21:36.091799 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:21:36.092956 (+ 1157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:21:36.092960 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:21:36.093416 (+ 456us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:21:36.093545 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:21:36.093552 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:21:36.094977 (+ 1425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:21:36.094993 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:21:36.095821 (+ 828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:21:36.095832 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:21:36.096626 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:21:36.096631 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:21:36.096891 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:21:36.097474 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:21:36.097493 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:21:36.099769 (+ 2276us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:21:36.102966 (+ 3197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:21:36.102972 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:21:36.102976 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:21:36.103362 (+ 386us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:21:36.103724 (+ 362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:21:36.103731 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:21:36.103734 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:21:36.103871 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:21:36.104520 (+ 649us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:21:36.104525 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:21:36.104686 (+ 161us) client_negotiation.cc:769] Sending connection context
0620 20:21:36.104858 (+ 172us) client_negotiation.cc:240] Negotiation successful
0620 20:21:36.105145 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57,"mutex_wait_us":19}
I20260620 20:21:36.105638 7330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:36.091815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46122 (local address 127.16.217.61:36701)
0620 20:21:36.092450 (+ 635us) server_negotiation.cc:206] Beginning negotiation
0620 20:21:36.092455 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:21:36.092505 (+ 50us) server_negotiation.cc:407] Connection header received
0620 20:21:36.092548 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:21:36.092554 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:21:36.092718 (+ 164us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:21:36.092970 (+ 252us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:21:36.093711 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:36.094847 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:36.095951 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:36.096476 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:36.096617 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:21:36.099971 (+ 3354us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:21:36.099987 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:21:36.100000 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:21:36.100031 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:21:36.102819 (+ 2788us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:21:36.103489 (+ 670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:21:36.103498 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:21:36.103502 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:21:36.103622 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:21:36.103969 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:21:36.103977 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:21:36.103987 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:21:36.104393 (+ 406us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:21:36.104609 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0620 20:21:36.104979 (+ 370us) server_negotiation.cc:299] Negotiation successful
0620 20:21:36.105250 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":81,"server-negotiator.queue_time_us":496,"thread_start_us":358,"threads_started":1}
I20260620 20:21:36.108065 7326 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.217.60" port: 39253 } has no permanent_uuid. Determining permanent_uuid...
I20260620 20:21:36.109334 7329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:36.108761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.60:39253 (local address 127.0.0.1:37194)
0620 20:21:36.108969 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0620 20:21:36.109107 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.217.60:39253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
W20260620 20:21:36.109661 7276 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.60:39253: connect: Connection refused (error 111)
I20260620 20:21:36.111330 7329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:36.110952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.60:39253 (local address 127.0.0.1:37204)
0620 20:21:36.111089 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0620 20:21:36.111222 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.217.60:39253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260620 20:21:36.111603 7326 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.217.60:39253: Network error: Client connection negotiation failed: client connection to 127.16.217.60:39253: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.217.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.60.keytab.
Entry for principal kudu/127.16.217.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.60.keytab.
I20260620 20:21:36.118809 17252 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
/tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986536687152-17252-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986536687152-17252-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986536687152-17252-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986536687152-17252-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.217.60:39253
--webserver_interface=127.16.217.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.60.keytab
--principal=kudu/127.16.217.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.217.20:45927
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.217.62:42485,127.16.217.61:36701,127.16.217.60:39253
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986536687152-17252-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986536687152-17252-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986536687152-17252-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 20:21:36.160945 7326 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.217.60" port: 39253 } attempt: 1
I20260620 20:21:36.162263 7329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:36.161656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.60:39253 (local address 127.0.0.1:37220)
0620 20:21:36.161917 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0620 20:21:36.162058 (+ 141us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.217.60:39253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260620 20:21:36.163364 7329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:36.162934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.60:39253 (local address 127.0.0.1:37236)
0620 20:21:36.163120 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0620 20:21:36.163217 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.217.60:39253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260620 20:21:36.163615 7326 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.217.60:39253: Network error: Client connection negotiation failed: client connection to 127.16.217.60:39253: connect: Connection refused (error 111)
I20260620 20:21:36.231853 7326 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.217.60" port: 39253 } attempt: 2
I20260620 20:21:36.233108 7329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:36.232544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.60:39253 (local address 127.0.0.1:37240)
0620 20:21:36.232774 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0620 20:21:36.232886 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.217.60:39253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260620 20:21:36.234249 7329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:36.233812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.60:39253 (local address 127.0.0.1:37252)
0620 20:21:36.234013 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0620 20:21:36.234098 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.217.60:39253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
W20260620 20:21:36.234489 7326 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.217.60:39253: Network error: Client connection negotiation failed: client connection to 127.16.217.60:39253: connect: Connection refused (error 111)
I20260620 20:21:36.250697 7252 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.217.61" port: 36701 } attempt: 5
Jun 20 20:21:36 dist-test-slave-2hpg krb5kdc[7172](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986895, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.217.62@KRBTEST.COM for kudu/127.16.217.61@KRBTEST.COM
I20260620 20:21:36.265647 7330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:36.251604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46128 (local address 127.16.217.61:36701)
0620 20:21:36.251780 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0620 20:21:36.251785 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:21:36.251805 (+ 20us) server_negotiation.cc:407] Connection header received
0620 20:21:36.252017 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:21:36.252021 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:21:36.252085 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:21:36.252170 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:21:36.253161 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:36.253856 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:36.255656 (+ 1800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:36.256107 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:36.256203 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:21:36.260232 (+ 4029us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:21:36.260237 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:21:36.260240 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:21:36.260273 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:21:36.263010 (+ 2737us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:21:36.263757 (+ 747us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:21:36.263761 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:21:36.263764 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:21:36.263880 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:21:36.264288 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:21:36.264291 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:21:36.264293 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:21:36.264569 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:21:36.264678 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0620 20:21:36.265248 (+ 570us) server_negotiation.cc:299] Negotiation successful
0620 20:21:36.265411 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 20:21:36.265996 7255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:36.251413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.61:36701 (local address 127.0.0.1:46128)
0620 20:21:36.251673 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0620 20:21:36.251699 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0620 20:21:36.251853 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:21:36.252237 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:21:36.252243 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:21:36.252736 (+ 493us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:21:36.253014 (+ 278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:21:36.253029 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:21:36.254047 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:21:36.254053 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:21:36.255503 (+ 1450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:21:36.255518 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:21:36.256252 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:21:36.256258 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:21:36.256496 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:21:36.257136 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:21:36.257159 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:21:36.260023 (+ 2864us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:21:36.263141 (+ 3118us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:21:36.263146 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:21:36.263152 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:21:36.263641 (+ 489us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:21:36.264006 (+ 365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:21:36.264010 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:21:36.264014 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:21:36.264175 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:21:36.264698 (+ 523us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:21:36.264706 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:21:36.264950 (+ 244us) client_negotiation.cc:769] Sending connection context
0620 20:21:36.265157 (+ 207us) client_negotiation.cc:240] Negotiation successful
0620 20:21:36.265501 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":80}
I20260620 20:21:36.267171 7252 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.217.60" port: 39253 } has no permanent_uuid. Determining permanent_uuid...
I20260620 20:21:36.268393 7255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:36.268014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.60:39253 (local address 127.0.0.1:37262)
0620 20:21:36.268182 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0620 20:21:36.268259 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.217.60:39253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260620 20:21:36.269527 7255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:36.269143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.60:39253 (local address 127.0.0.1:37268)
0620 20:21:36.269322 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0620 20:21:36.269415 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.217.60:39253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
W20260620 20:21:36.269851 7252 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.217.60:39253: Network error: Client connection negotiation failed: client connection to 127.16.217.60:39253: connect: Connection refused (error 111)
I20260620 20:21:36.321159 7252 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.217.60" port: 39253 } attempt: 1
I20260620 20:21:36.322494 7255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:36.321979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.60:39253 (local address 127.0.0.1:37276)
0620 20:21:36.322190 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0620 20:21:36.322302 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.217.60:39253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260620 20:21:36.323526 7255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:36.323190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.60:39253 (local address 127.0.0.1:37280)
0620 20:21:36.323341 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0620 20:21:36.323417 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.217.60:39253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260620 20:21:36.323765 7252 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.217.60:39253: Network error: Client connection negotiation failed: client connection to 127.16.217.60:39253: connect: Connection refused (error 111)
I20260620 20:21:36.325717 7326 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.217.60" port: 39253 } attempt: 3
I20260620 20:21:36.326758 7329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:36.326293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.60:39253 (local address 127.0.0.1:37286)
0620 20:21:36.326487 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0620 20:21:36.326586 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.217.60:39253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260620 20:21:36.327870 7329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:36.327406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.60:39253 (local address 127.0.0.1:37292)
0620 20:21:36.327642 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0620 20:21:36.327729 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.217.60:39253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136}
W20260620 20:21:36.328104 7326 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.217.60:39253: Network error: Client connection negotiation failed: client connection to 127.16.217.60:39253: connect: Connection refused (error 111)
I20260620 20:21:36.392007 7252 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.217.60" port: 39253 } attempt: 2
I20260620 20:21:36.393225 7255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:36.392707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.60:39253 (local address 127.0.0.1:37300)
0620 20:21:36.392958 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0620 20:21:36.393052 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.217.60:39253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260620 20:21:36.394367 7255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:36.394009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.60:39253 (local address 127.0.0.1:37316)
0620 20:21:36.394156 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0620 20:21:36.394254 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.217.60:39253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260620 20:21:36.394613 7252 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.217.60:39253: Network error: Client connection negotiation failed: client connection to 127.16.217.60:39253: connect: Connection refused (error 111)
W20260620 20:21:36.466243 7333 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 20:21:36.466558 7333 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:21:36.466701 7333 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:21:36.466833 7333 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260620 20:21:36.471320 7326 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.217.60" port: 39253 } attempt: 4
I20260620 20:21:36.472510 7329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:36.471960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.60:39253 (local address 127.0.0.1:37330)
0620 20:21:36.472215 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0620 20:21:36.472318 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.217.60:39253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260620 20:21:36.473503 7329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:36.473159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.60:39253 (local address 127.0.0.1:37344)
0620 20:21:36.473300 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0620 20:21:36.473381 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.217.60:39253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260620 20:21:36.473712 7326 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.217.60:39253: Network error: Client connection negotiation failed: client connection to 127.16.217.60:39253: connect: Connection refused (error 111)
W20260620 20:21:36.475013 7333 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:21:36.475085 7333 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:21:36.475142 7333 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:21:36.475191 7333 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:21:36.475245 7333 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:21:36.485116 7333 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.217.20:45927
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986536687152-17252-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986536687152-17252-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.16.217.62:42485,127.16.217.61:36701,127.16.217.60:39253
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.60.keytab
--principal=kudu/127.16.217.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.217.60:39253
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986536687152-17252-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.16.217.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986536687152-17252-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:21:36.486447 7333 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:21:36.487818 7333 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:21:36.500028 7339 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:21:36.500154 7338 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:21:36.500092 7341 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:21:36.500854 7252 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.217.60" port: 39253 } attempt: 3
I20260620 20:21:36.501327 7333 server_base.cc:1034] running on GCE node
I20260620 20:21:36.502108 7255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:36.501540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.60:39253 (local address 127.0.0.1:37354)
0620 20:21:36.501786 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0620 20:21:36.501927 (+ 141us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.217.60:39253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260620 20:21:36.502229 7333 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:21:36.503293 7255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:36.502834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.60:39253 (local address 127.0.0.1:37360)
0620 20:21:36.503073 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0620 20:21:36.503164 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.217.60:39253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107}
W20260620 20:21:36.503516 7252 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.217.60:39253: Network error: Client connection negotiation failed: client connection to 127.16.217.60:39253: connect: Connection refused (error 111)
I20260620 20:21:36.503820 7333 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:21:36.505079 7333 hybrid_clock.cc:648] HybridClock initialized: now 1781986896504997 us; error 36 us; skew 500 ppm
I20260620 20:21:36.505431 7333 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:21:36 dist-test-slave-2hpg krb5kdc[7172](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986896, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.217.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:21:36.508833 7333 init.cc:377] Logged in from keytab as kudu/127.16.217.60@KRBTEST.COM (short username kudu)
I20260620 20:21:36.510318 7333 webserver.cc:460] Webserver started at http://127.16.217.60:40585/ using document root <none> and password file <none>
I20260620 20:21:36.511029 7333 fs_manager.cc:359] Metadata directory not provided
I20260620 20:21:36.511116 7333 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:21:36.511436 7333 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:21:36.514477 7333 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986536687152-17252-0/minicluster-data/master-2/data/instance:
uuid: "1513f03de35d4f828fa4cfe4501126d1"
format_stamp: "Formatted at 2026-06-20 20:21:36 on dist-test-slave-2hpg"
server_key: "1028428509b423f73b9e0072a3af4def"
server_key_iv: "0e5ed7e633d52c7ce00b055f7211a7f6"
server_key_version: "encryptionkey@0"
I20260620 20:21:36.515262 7333 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986536687152-17252-0/minicluster-data/master-2/wal/instance:
uuid: "1513f03de35d4f828fa4cfe4501126d1"
format_stamp: "Formatted at 2026-06-20 20:21:36 on dist-test-slave-2hpg"
server_key: "1028428509b423f73b9e0072a3af4def"
server_key_iv: "0e5ed7e633d52c7ce00b055f7211a7f6"
server_key_version: "encryptionkey@0"
I20260620 20:21:36.520310 7333 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.001s
I20260620 20:21:36.523854 7348 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:21:36.525277 7333 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260620 20:21:36.525472 7333 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986536687152-17252-0/minicluster-data/master-2/data,/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986536687152-17252-0/minicluster-data/master-2/wal
uuid: "1513f03de35d4f828fa4cfe4501126d1"
format_stamp: "Formatted at 2026-06-20 20:21:36 on dist-test-slave-2hpg"
server_key: "1028428509b423f73b9e0072a3af4def"
server_key_iv: "0e5ed7e633d52c7ce00b055f7211a7f6"
server_key_version: "encryptionkey@0"
I20260620 20:21:36.525688 7333 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986536687152-17252-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986536687152-17252-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986536687152-17252-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:21:36.540316 7333 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:21:36.544667 7333 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:21:36.544950 7333 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:21:36.564942 7333 rpc_server.cc:307] RPC server started. Bound to: 127.16.217.60:39253
I20260620 20:21:36.564985 7399 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.217.60:39253 every 8 connection(s)
I20260620 20:21:36.566371 7333 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986536687152-17252-0/minicluster-data/master-2/data/info.pb
I20260620 20:21:36.570106 7400 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:21:36.574242 17252 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu as pid 7333
I20260620 20:21:36.574424 17252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986536687152-17252-0/minicluster-data/master-2/wal/instance
I20260620 20:21:36.574847 17252 external_mini_cluster.cc:1428] Setting key 3a0268af239e09dd11b42a58898567c5
I20260620 20:21:36.577003 7400 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.217.62" port: 42485 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 20:21:36 dist-test-slave-2hpg krb5kdc[7172](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986896, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.217.60@KRBTEST.COM for kudu/127.16.217.62@KRBTEST.COM
Jun 20 20:21:36 dist-test-slave-2hpg krb5kdc[7172](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986895, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.217.60@KRBTEST.COM
I20260620 20:21:36.595676 7256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:36.579666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48842 (local address 127.16.217.62:42485)
0620 20:21:36.579879 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0620 20:21:36.579884 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:21:36.580583 (+ 699us) server_negotiation.cc:407] Connection header received
0620 20:21:36.580744 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:21:36.580749 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:21:36.580837 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:21:36.580951 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:21:36.582535 (+ 1584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:36.583311 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:36.584875 (+ 1564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:36.585374 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:36.585478 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:21:36.589320 (+ 3842us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:21:36.589330 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:21:36.589335 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:21:36.589377 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:21:36.592444 (+ 3067us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:21:36.593184 (+ 740us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:21:36.593189 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:21:36.593193 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:21:36.593345 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:21:36.593772 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:21:36.593778 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:21:36.593781 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:21:36.594119 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:21:36.594287 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0620 20:21:36.595006 (+ 719us) server_negotiation.cc:299] Negotiation successful
0620 20:21:36.595283 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260620 20:21:36.596705 7403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:36.576633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37368 (local address 127.16.217.60:39253)
0620 20:21:36.577405 (+ 772us) server_negotiation.cc:206] Beginning negotiation
0620 20:21:36.577417 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 20:21:36.577460 (+ 43us) server_negotiation.cc:407] Connection header received
0620 20:21:36.579183 (+ 1723us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:21:36.579207 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:21:36.579715 (+ 508us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:21:36.580148 (+ 433us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:21:36.582371 (+ 2223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:36.583681 (+ 1310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:36.585011 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:36.585542 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:36.585781 (+ 239us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:21:36.589967 (+ 4186us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:21:36.589975 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:21:36.589986 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:21:36.590013 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:21:36.592864 (+ 2851us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:21:36.593566 (+ 702us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:21:36.593584 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:21:36.593587 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:21:36.593714 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:21:36.594116 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:21:36.594122 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:21:36.594125 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:21:36.594611 (+ 486us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:21:36.594946 (+ 335us) server_negotiation.cc:1035] Waiting for connection context
0620 20:21:36.595263 (+ 317us) server_negotiation.cc:299] Negotiation successful
0620 20:21:36.595585 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":69,"server-negotiator.queue_time_us":558,"thread_start_us":361,"threads_started":1}
I20260620 20:21:36.596733 7404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:36.579512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:42485 (local address 127.0.0.1:48842)
0620 20:21:36.580426 (+ 914us) negotiation.cc:107] Waiting for socket to connect
0620 20:21:36.580465 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 20:21:36.580614 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:21:36.581023 (+ 409us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:21:36.581028 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:21:36.581635 (+ 607us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:21:36.582270 (+ 635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:21:36.582316 (+ 46us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:21:36.583466 (+ 1150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:21:36.583471 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:21:36.584719 (+ 1248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:21:36.584730 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:21:36.585636 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:21:36.585642 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:21:36.585932 (+ 290us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:21:36.586804 (+ 872us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:21:36.586824 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:21:36.589112 (+ 2288us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:21:36.592603 (+ 3491us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:21:36.592614 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:21:36.592626 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:21:36.593060 (+ 434us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:21:36.593465 (+ 405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:21:36.593470 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:21:36.593474 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:21:36.593659 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:21:36.594255 (+ 596us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:21:36.594263 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:21:36.594630 (+ 367us) client_negotiation.cc:769] Sending connection context
0620 20:21:36.594924 (+ 294us) client_negotiation.cc:240] Negotiation successful
0620 20:21:36.595450 (+ 526us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":670,"mutex_wait_us":111,"thread_start_us":533,"threads_started":1}
I20260620 20:21:36.598380 7400 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.217.61" port: 36701 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 20:21:36 dist-test-slave-2hpg krb5kdc[7172](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986896, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.217.60@KRBTEST.COM for kudu/127.16.217.61@KRBTEST.COM
Jun 20 20:21:36 dist-test-slave-2hpg krb5kdc[7172](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986895, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.217.62@KRBTEST.COM
I20260620 20:21:36.611780 7404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:36.599001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.61:36701 (local address 127.0.0.1:46138)
0620 20:21:36.599438 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0620 20:21:36.599457 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 20:21:36.599564 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:21:36.599944 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:21:36.599948 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:21:36.600267 (+ 319us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:21:36.600408 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:21:36.600415 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:21:36.601396 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:21:36.601404 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:21:36.602382 (+ 978us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:21:36.602393 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:21:36.603104 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:21:36.603110 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:21:36.603314 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:21:36.603874 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:21:36.603891 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:21:36.606219 (+ 2328us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:21:36.609363 (+ 3144us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:21:36.609371 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:21:36.609375 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:21:36.609730 (+ 355us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:21:36.610161 (+ 431us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:21:36.610166 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:21:36.610169 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:21:36.610316 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:21:36.610840 (+ 524us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:21:36.610845 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:21:36.610998 (+ 153us) client_negotiation.cc:769] Sending connection context
0620 20:21:36.611133 (+ 135us) client_negotiation.cc:240] Negotiation successful
0620 20:21:36.611411 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":340}
I20260620 20:21:36.611816 7330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:36.599254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46138 (local address 127.16.217.61:36701)
0620 20:21:36.599433 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0620 20:21:36.599438 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:21:36.599505 (+ 67us) server_negotiation.cc:407] Connection header received
0620 20:21:36.599714 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:21:36.599720 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:21:36.599808 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:21:36.599909 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:21:36.600558 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:36.601259 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:36.602537 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:36.602956 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:36.603097 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:21:36.606397 (+ 3300us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:21:36.606408 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:21:36.606411 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:21:36.606442 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:21:36.609237 (+ 2795us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:21:36.609842 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:21:36.609849 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:21:36.609853 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:21:36.610042 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:21:36.610434 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:21:36.610440 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:21:36.610443 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:21:36.610724 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:21:36.610880 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0620 20:21:36.611232 (+ 352us) server_negotiation.cc:299] Negotiation successful
0620 20:21:36.611441 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 20:21:36.612670 7400 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.217.60" port: 39253 } has no permanent_uuid. Determining permanent_uuid...
I20260620 20:21:36.614850 7256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:36.599735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48846 (local address 127.16.217.62:42485)
0620 20:21:36.600158 (+ 423us) server_negotiation.cc:206] Beginning negotiation
0620 20:21:36.600163 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:21:36.600178 (+ 15us) server_negotiation.cc:407] Connection header received
0620 20:21:36.600225 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:21:36.600228 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:21:36.600275 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:21:36.600390 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:21:36.601494 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:36.602283 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:36.603485 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:36.603884 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:36.604020 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:21:36.608867 (+ 4847us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:21:36.608871 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:21:36.608874 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:21:36.608904 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:21:36.611538 (+ 2634us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:21:36.612489 (+ 951us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:21:36.612494 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:21:36.612499 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:21:36.612645 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:21:36.613050 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:21:36.613055 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:21:36.613060 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:21:36.613464 (+ 404us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:21:36.613738 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0620 20:21:36.614279 (+ 541us) server_negotiation.cc:299] Negotiation successful
0620 20:21:36.614516 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":301}
Jun 20 20:21:36 dist-test-slave-2hpg krb5kdc[7172](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986896, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.217.60@KRBTEST.COM for kudu/127.16.217.60@KRBTEST.COM
Jun 20 20:21:36 dist-test-slave-2hpg krb5kdc[7172](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986895, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.217.61@KRBTEST.COM
I20260620 20:21:36.627321 7404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:36.613591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.60:39253 (local address 127.0.0.1:37376)
0620 20:21:36.613711 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0620 20:21:36.613727 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 20:21:36.613847 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:21:36.614167 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:21:36.614171 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:21:36.614532 (+ 361us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:21:36.614777 (+ 245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:21:36.614787 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:21:36.615952 (+ 1165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:21:36.615960 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:21:36.617155 (+ 1195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:21:36.617169 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:21:36.617833 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:21:36.617842 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:21:36.618082 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:21:36.618712 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:21:36.618727 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:21:36.620875 (+ 2148us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:21:36.624665 (+ 3790us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:21:36.624669 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:21:36.624671 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:21:36.625023 (+ 352us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:21:36.625406 (+ 383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:21:36.625411 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:21:36.625414 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:21:36.625577 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:21:36.626146 (+ 569us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:21:36.626150 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:21:36.626327 (+ 177us) client_negotiation.cc:769] Sending connection context
0620 20:21:36.626505 (+ 178us) client_negotiation.cc:240] Negotiation successful
0620 20:21:36.626860 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33,"mutex_wait_us":1}
I20260620 20:21:36.627413 7403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:36.613448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37376 (local address 127.16.217.60:39253)
0620 20:21:36.613649 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0620 20:21:36.613655 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:21:36.613796 (+ 141us) server_negotiation.cc:407] Connection header received
0620 20:21:36.613986 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:21:36.613991 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:21:36.614053 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:21:36.614147 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:21:36.614897 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:36.615842 (+ 945us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:36.617291 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:36.617686 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:36.617780 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:21:36.621063 (+ 3283us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:21:36.621067 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:21:36.621070 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:21:36.621096 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:21:36.624531 (+ 3435us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:21:36.625147 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:21:36.625155 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:21:36.625160 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:21:36.625291 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:21:36.625680 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:21:36.625688 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:21:36.625694 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:21:36.626036 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:21:36.626181 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0620 20:21:36.626580 (+ 399us) server_negotiation.cc:299] Negotiation successful
0620 20:21:36.626963 (+ 383us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":199,"server-negotiator.queue_time_us":88}
I20260620 20:21:36.632088 7330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:36.616523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46144 (local address 127.16.217.61:36701)
0620 20:21:36.616675 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0620 20:21:36.616679 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:21:36.616694 (+ 15us) server_negotiation.cc:407] Connection header received
0620 20:21:36.616783 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:21:36.616786 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:21:36.616837 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:21:36.616908 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:21:36.618356 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:36.619161 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:36.620590 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:36.621167 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:36.621266 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:21:36.626346 (+ 5080us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:21:36.626353 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:21:36.626357 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:21:36.626395 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:21:36.629554 (+ 3159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:21:36.630350 (+ 796us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:21:36.630354 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:21:36.630356 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:21:36.630496 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:21:36.630881 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:21:36.630885 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:21:36.630887 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:21:36.631140 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:21:36.631333 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0620 20:21:36.631626 (+ 293us) server_negotiation.cc:299] Negotiation successful
0620 20:21:36.631790 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 20:21:36.633214 7400 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1513f03de35d4f828fa4cfe4501126d1: Bootstrap starting.
I20260620 20:21:36.636862 7400 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1513f03de35d4f828fa4cfe4501126d1: Neither blocks nor log segments found. Creating new log.
I20260620 20:21:36.637084 17252 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 20:21:36.638016 7400 log.cc:824] T 00000000000000000000000000000000 P 1513f03de35d4f828fa4cfe4501126d1: Log is configured to *not* fsync() on all Append() calls
I20260620 20:21:36.640707 7400 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1513f03de35d4f828fa4cfe4501126d1: No bootstrap required, opened a new log
I20260620 20:21:36.639787 7227 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "f180af9b6d7043d7a7ba302ffedeeb3a"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48846
I20260620 20:21:36.647996 7400 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1513f03de35d4f828fa4cfe4501126d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f180af9b6d7043d7a7ba302ffedeeb3a" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 42485 } } peers { permanent_uuid: "b8e62a119beb46cea5763e672f95303a" member_type: VOTER last_known_addr { host: "127.16.217.61" port: 36701 } } peers { permanent_uuid: "1513f03de35d4f828fa4cfe4501126d1" member_type: VOTER last_known_addr { host: "127.16.217.60" port: 39253 } }
I20260620 20:21:36.648347 7400 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1513f03de35d4f828fa4cfe4501126d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1513f03de35d4f828fa4cfe4501126d1, State: Initialized, Role: FOLLOWER
I20260620 20:21:36.648790 7400 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1513f03de35d4f828fa4cfe4501126d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f180af9b6d7043d7a7ba302ffedeeb3a" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 42485 } } peers { permanent_uuid: "b8e62a119beb46cea5763e672f95303a" member_type: VOTER last_known_addr { host: "127.16.217.61" port: 36701 } } peers { permanent_uuid: "1513f03de35d4f828fa4cfe4501126d1" member_type: VOTER last_known_addr { host: "127.16.217.60" port: 39253 } }
I20260620 20:21:36.649772 7406 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1513f03de35d4f828fa4cfe4501126d1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f180af9b6d7043d7a7ba302ffedeeb3a" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 42485 } } peers { permanent_uuid: "b8e62a119beb46cea5763e672f95303a" member_type: VOTER last_known_addr { host: "127.16.217.61" port: 36701 } } peers { permanent_uuid: "1513f03de35d4f828fa4cfe4501126d1" member_type: VOTER last_known_addr { host: "127.16.217.60" port: 39253 } } }
I20260620 20:21:36.650229 7406 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1513f03de35d4f828fa4cfe4501126d1 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 20:21:36.650768 7400 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1513f03de35d4f828fa4cfe4501126d1 [sys.catalog]: configured and running, proceeding with master startup.
W20260620 20:21:36.660862 7417 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1513f03de35d4f828fa4cfe4501126d1: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 20:21:36.661013 7417 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 20:21:36.658722 7333 master_runner.cc:186] Error getting master registration for 127.16.217.62:42485: OK, instance_id { permanent_uuid: "f180af9b6d7043d7a7ba302ffedeeb3a" instance_seqno: 1781986895536509 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 20:21:36.667246 7333 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 20:21:36.680744 7252 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.217.60" port: 39253 } attempt: 4
Jun 20 20:21:36 dist-test-slave-2hpg krb5kdc[7172](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986895, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.217.62@KRBTEST.COM for kudu/127.16.217.60@KRBTEST.COM
I20260620 20:21:36.699442 7255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:36.681551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.60:39253 (local address 127.0.0.1:37388)
0620 20:21:36.681852 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0620 20:21:36.681913 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0620 20:21:36.682043 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:21:36.682409 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:21:36.682414 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:21:36.682835 (+ 421us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:21:36.683011 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:21:36.683020 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:21:36.684071 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:21:36.684078 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:21:36.685062 (+ 984us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:21:36.685073 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:21:36.685905 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:21:36.685911 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:21:36.686153 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:21:36.689953 (+ 3800us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:21:36.689981 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:21:36.692671 (+ 2690us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:21:36.696906 (+ 4235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:21:36.696911 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:21:36.696916 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:21:36.697284 (+ 368us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:21:36.697673 (+ 389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:21:36.697677 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:21:36.697681 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:21:36.697822 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:21:36.698441 (+ 619us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:21:36.698446 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:21:36.698622 (+ 176us) client_negotiation.cc:769] Sending connection context
0620 20:21:36.698758 (+ 136us) client_negotiation.cc:240] Negotiation successful
0620 20:21:36.699055 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163}
I20260620 20:21:36.699445 7403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:36.681524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37388 (local address 127.16.217.60:39253)
0620 20:21:36.681742 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0620 20:21:36.681749 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:21:36.682000 (+ 251us) server_negotiation.cc:407] Connection header received
0620 20:21:36.682196 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:21:36.682200 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:21:36.682266 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:21:36.682376 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:21:36.683154 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:36.683913 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:36.685207 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:36.685697 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:36.685901 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:21:36.692868 (+ 6967us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:21:36.692875 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:21:36.692884 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:21:36.692931 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:21:36.696773 (+ 3842us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:21:36.697408 (+ 635us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:21:36.697414 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:21:36.697418 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:21:36.697556 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:21:36.697975 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:21:36.697983 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:21:36.697986 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:21:36.698305 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:21:36.698503 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0620 20:21:36.698862 (+ 359us) server_negotiation.cc:299] Negotiation successful
0620 20:21:36.699068 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 20:21:36.704144 7252 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f180af9b6d7043d7a7ba302ffedeeb3a: Bootstrap starting.
I20260620 20:21:36.707892 7252 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f180af9b6d7043d7a7ba302ffedeeb3a: Neither blocks nor log segments found. Creating new log.
I20260620 20:21:36.709187 7252 log.cc:824] T 00000000000000000000000000000000 P f180af9b6d7043d7a7ba302ffedeeb3a: Log is configured to *not* fsync() on all Append() calls
I20260620 20:21:36.711907 7252 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f180af9b6d7043d7a7ba302ffedeeb3a: No bootstrap required, opened a new log
I20260620 20:21:36.719419 7252 raft_consensus.cc:348] T 00000000000000000000000000000000 P f180af9b6d7043d7a7ba302ffedeeb3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f180af9b6d7043d7a7ba302ffedeeb3a" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 42485 } } peers { permanent_uuid: "b8e62a119beb46cea5763e672f95303a" member_type: VOTER last_known_addr { host: "127.16.217.61" port: 36701 } } peers { permanent_uuid: "1513f03de35d4f828fa4cfe4501126d1" member_type: VOTER last_known_addr { host: "127.16.217.60" port: 39253 } }
I20260620 20:21:36.719825 7252 raft_consensus.cc:729] T 00000000000000000000000000000000 P f180af9b6d7043d7a7ba302ffedeeb3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f180af9b6d7043d7a7ba302ffedeeb3a, State: Initialized, Role: FOLLOWER
I20260620 20:21:36.720261 7252 consensus_queue.cc:260] T 00000000000000000000000000000000 P f180af9b6d7043d7a7ba302ffedeeb3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f180af9b6d7043d7a7ba302ffedeeb3a" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 42485 } } peers { permanent_uuid: "b8e62a119beb46cea5763e672f95303a" member_type: VOTER last_known_addr { host: "127.16.217.61" port: 36701 } } peers { permanent_uuid: "1513f03de35d4f828fa4cfe4501126d1" member_type: VOTER last_known_addr { host: "127.16.217.60" port: 39253 } }
I20260620 20:21:36.721287 7420 sys_catalog.cc:455] T 00000000000000000000000000000000 P f180af9b6d7043d7a7ba302ffedeeb3a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f180af9b6d7043d7a7ba302ffedeeb3a" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 42485 } } peers { permanent_uuid: "b8e62a119beb46cea5763e672f95303a" member_type: VOTER last_known_addr { host: "127.16.217.61" port: 36701 } } peers { permanent_uuid: "1513f03de35d4f828fa4cfe4501126d1" member_type: VOTER last_known_addr { host: "127.16.217.60" port: 39253 } } }
I20260620 20:21:36.721633 7420 sys_catalog.cc:458] T 00000000000000000000000000000000 P f180af9b6d7043d7a7ba302ffedeeb3a [sys.catalog]: This master's current role is: FOLLOWER
I20260620 20:21:36.722448 7252 sys_catalog.cc:564] T 00000000000000000000000000000000 P f180af9b6d7043d7a7ba302ffedeeb3a [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:21:36.722601 7227 raft_consensus.cc:482] T 00000000000000000000000000000000 P f180af9b6d7043d7a7ba302ffedeeb3a [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260620 20:21:36.722802 7227 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f180af9b6d7043d7a7ba302ffedeeb3a [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:21:36.726377 7227 raft_consensus.cc:504] T 00000000000000000000000000000000 P f180af9b6d7043d7a7ba302ffedeeb3a [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f180af9b6d7043d7a7ba302ffedeeb3a" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 42485 } } peers { permanent_uuid: "b8e62a119beb46cea5763e672f95303a" member_type: VOTER last_known_addr { host: "127.16.217.61" port: 36701 } } peers { permanent_uuid: "1513f03de35d4f828fa4cfe4501126d1" member_type: VOTER last_known_addr { host: "127.16.217.60" port: 39253 } }
I20260620 20:21:36.728883 7375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f180af9b6d7043d7a7ba302ffedeeb3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "1513f03de35d4f828fa4cfe4501126d1"
I20260620 20:21:36.729364 7375 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1513f03de35d4f828fa4cfe4501126d1 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:21:36.732844 7375 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 1513f03de35d4f828fa4cfe4501126d1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f180af9b6d7043d7a7ba302ffedeeb3a in term 1.
I20260620 20:21:36.733889 7227 leader_election.cc:288] T 00000000000000000000000000000000 P f180af9b6d7043d7a7ba302ffedeeb3a [CANDIDATE]: Term 1 election: Requested vote from peers b8e62a119beb46cea5763e672f95303a (127.16.217.61:36701), 1513f03de35d4f828fa4cfe4501126d1 (127.16.217.60:39253)
I20260620 20:21:36.734033 7202 leader_election.cc:302] T 00000000000000000000000000000000 P f180af9b6d7043d7a7ba302ffedeeb3a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1513f03de35d4f828fa4cfe4501126d1, f180af9b6d7043d7a7ba302ffedeeb3a; no voters:
I20260620 20:21:36.734522 7420 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f180af9b6d7043d7a7ba302ffedeeb3a [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:21:36.735198 7420 raft_consensus.cc:686] T 00000000000000000000000000000000 P f180af9b6d7043d7a7ba302ffedeeb3a [term 1 LEADER]: Becoming Leader. State: Replica: f180af9b6d7043d7a7ba302ffedeeb3a, State: Running, Role: LEADER
I20260620 20:21:36.735739 7420 consensus_queue.cc:237] T 00000000000000000000000000000000 P f180af9b6d7043d7a7ba302ffedeeb3a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f180af9b6d7043d7a7ba302ffedeeb3a" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 42485 } } peers { permanent_uuid: "b8e62a119beb46cea5763e672f95303a" member_type: VOTER last_known_addr { host: "127.16.217.61" port: 36701 } } peers { permanent_uuid: "1513f03de35d4f828fa4cfe4501126d1" member_type: VOTER last_known_addr { host: "127.16.217.60" port: 39253 } }
I20260620 20:21:36.748018 7431 sys_catalog.cc:455] T 00000000000000000000000000000000 P f180af9b6d7043d7a7ba302ffedeeb3a [sys.catalog]: SysCatalogTable state changed. Reason: New leader f180af9b6d7043d7a7ba302ffedeeb3a. Latest consensus state: current_term: 1 leader_uuid: "f180af9b6d7043d7a7ba302ffedeeb3a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f180af9b6d7043d7a7ba302ffedeeb3a" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 42485 } } peers { permanent_uuid: "b8e62a119beb46cea5763e672f95303a" member_type: VOTER last_known_addr { host: "127.16.217.61" port: 36701 } } peers { permanent_uuid: "1513f03de35d4f828fa4cfe4501126d1" member_type: VOTER last_known_addr { host: "127.16.217.60" port: 39253 } } }
I20260620 20:21:36.748430 7431 sys_catalog.cc:458] T 00000000000000000000000000000000 P f180af9b6d7043d7a7ba302ffedeeb3a [sys.catalog]: This master's current role is: LEADER
I20260620 20:21:36.756609 7437 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 20:21:36.758275 7440 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260620 20:21:36.772920 7326 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.217.60" port: 39253 } attempt: 5
W20260620 20:21:36.774910 7206 tablet.cc:2369] T 00000000000000000000000000000000 P f180af9b6d7043d7a7ba302ffedeeb3a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 20:21:36.780736 7403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:36.750254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37394 (local address 127.16.217.60:39253)
0620 20:21:36.750474 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0620 20:21:36.750481 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:21:36.750819 (+ 338us) server_negotiation.cc:407] Connection header received
0620 20:21:36.751347 (+ 528us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:21:36.751356 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:21:36.751464 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:21:36.751539 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:21:36.756986 (+ 5447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:36.758492 (+ 1506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:36.762315 (+ 3823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:36.763056 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:36.763183 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:21:36.769591 (+ 6408us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:21:36.769599 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:21:36.769604 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:21:36.769652 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:21:36.773974 (+ 4322us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:21:36.775505 (+ 1531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:21:36.775511 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:21:36.775514 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:21:36.775685 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:21:36.776545 (+ 860us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:21:36.776553 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:21:36.776558 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:21:36.776986 (+ 428us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:21:36.777308 (+ 322us) server_negotiation.cc:1035] Waiting for connection context
0620 20:21:36.778678 (+ 1370us) server_negotiation.cc:299] Negotiation successful
0620 20:21:36.779213 (+ 535us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 20:21:36.763000 7185 master_runner.cc:186] Error getting master registration for 127.16.217.61:36701: OK, instance_id { permanent_uuid: "b8e62a119beb46cea5763e672f95303a" instance_seqno: 1781986896051098 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
Jun 20 20:21:36 dist-test-slave-2hpg krb5kdc[7172](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986895, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.217.61@KRBTEST.COM for kudu/127.16.217.60@KRBTEST.COM
I20260620 20:21:36.794236 7185 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260620 20:21:36.797240 7354 tablet.cc:2369] T 00000000000000000000000000000000 P 1513f03de35d4f828fa4cfe4501126d1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 20:21:36.800722 7256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:36.749571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48850 (local address 127.16.217.62:42485)
0620 20:21:36.749771 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0620 20:21:36.749780 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 20:21:36.766640 (+ 16860us) server_negotiation.cc:407] Connection header received
0620 20:21:36.766973 (+ 333us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:21:36.766982 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:21:36.767090 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:21:36.767179 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:21:36.769676 (+ 2497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:36.771437 (+ 1761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:36.786021 (+ 14584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:36.787041 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:36.787172 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:21:36.791751 (+ 4579us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:21:36.791761 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:21:36.791769 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:21:36.791824 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:21:36.796555 (+ 4731us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:21:36.797615 (+ 1060us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:21:36.797623 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:21:36.797627 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:21:36.797825 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:21:36.798321 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:21:36.798329 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:21:36.798335 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:21:36.798681 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:21:36.799190 (+ 509us) server_negotiation.cc:1035] Waiting for connection context
0620 20:21:36.799828 (+ 638us) server_negotiation.cc:299] Negotiation successful
0620 20:21:36.800090 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260620 20:21:36.801262 7437 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 20:21:36.807502 7329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:36.773710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.60:39253 (local address 127.0.0.1:37404)
0620 20:21:36.774111 (+ 401us) negotiation.cc:107] Waiting for socket to connect
0620 20:21:36.774133 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 20:21:36.774228 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:21:36.781953 (+ 7725us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:21:36.781959 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:21:36.782477 (+ 518us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:21:36.782774 (+ 297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:21:36.782791 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:21:36.785846 (+ 3055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:21:36.785854 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:21:36.787443 (+ 1589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:21:36.787458 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:21:36.788939 (+ 1481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:21:36.788945 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:21:36.789172 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:21:36.789927 (+ 755us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:21:36.789968 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:21:36.793516 (+ 3548us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:21:36.803581 (+ 10065us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:21:36.803586 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:21:36.803590 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:21:36.804173 (+ 583us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:21:36.804874 (+ 701us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:21:36.804880 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:21:36.804883 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:21:36.805079 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:21:36.806412 (+ 1333us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:21:36.806417 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:21:36.806617 (+ 200us) client_negotiation.cc:769] Sending connection context
0620 20:21:36.806781 (+ 164us) client_negotiation.cc:240] Negotiation successful
0620 20:21:36.807121 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267}
I20260620 20:21:36.809363 7403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:36.774432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37404 (local address 127.16.217.60:39253)
0620 20:21:36.781302 (+ 6870us) server_negotiation.cc:206] Beginning negotiation
0620 20:21:36.781310 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 20:21:36.781360 (+ 50us) server_negotiation.cc:407] Connection header received
0620 20:21:36.781455 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:21:36.781499 (+ 44us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:21:36.781658 (+ 159us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:21:36.782966 (+ 1308us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:21:36.783260 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:36.785597 (+ 2337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:36.787683 (+ 2086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:36.788724 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:36.789574 (+ 850us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:21:36.794126 (+ 4552us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:21:36.794134 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:21:36.794139 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:21:36.794202 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:21:36.803395 (+ 9193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:21:36.804378 (+ 983us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:21:36.804414 (+ 36us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:21:36.804417 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:21:36.804682 (+ 265us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:21:36.805280 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:21:36.805286 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:21:36.805317 (+ 31us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:21:36.806202 (+ 885us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:21:36.807849 (+ 1647us) server_negotiation.cc:1035] Waiting for connection context
0620 20:21:36.808121 (+ 272us) server_negotiation.cc:299] Negotiation successful
0620 20:21:36.808565 (+ 444us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6624,"thread_start_us":17128,"threads_started":1}
I20260620 20:21:36.818403 7326 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b8e62a119beb46cea5763e672f95303a: Bootstrap starting.
I20260620 20:21:36.824178 7330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:36.782300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46154 (local address 127.16.217.61:36701)
0620 20:21:36.782469 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0620 20:21:36.782474 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:21:36.791414 (+ 8940us) server_negotiation.cc:407] Connection header received
0620 20:21:36.791903 (+ 489us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:21:36.791912 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:21:36.792111 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:21:36.792200 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:21:36.794461 (+ 2261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:36.795574 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:36.803386 (+ 7812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:36.804146 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:36.804268 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:21:36.807883 (+ 3615us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:21:36.807891 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:21:36.807897 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:21:36.807933 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:21:36.811450 (+ 3517us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:21:36.812383 (+ 933us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:21:36.812388 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:21:36.812391 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:21:36.812545 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:21:36.822359 (+ 9814us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:21:36.822365 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:21:36.822369 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:21:36.822844 (+ 475us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:21:36.823202 (+ 358us) server_negotiation.cc:1035] Waiting for connection context
0620 20:21:36.823521 (+ 319us) server_negotiation.cc:299] Negotiation successful
0620 20:21:36.823805 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 20:21:36.834494 7326 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b8e62a119beb46cea5763e672f95303a: Neither blocks nor log segments found. Creating new log.
I20260620 20:21:36.836313 7326 log.cc:824] T 00000000000000000000000000000000 P b8e62a119beb46cea5763e672f95303a: Log is configured to *not* fsync() on all Append() calls
I20260620 20:21:36.840749 7326 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b8e62a119beb46cea5763e672f95303a: No bootstrap required, opened a new log
I20260620 20:21:36.842026 7375 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 1513f03de35d4f828fa4cfe4501126d1 [term 1 FOLLOWER]: Refusing update from remote peer f180af9b6d7043d7a7ba302ffedeeb3a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:21:36.842954 7431 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f180af9b6d7043d7a7ba302ffedeeb3a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1513f03de35d4f828fa4cfe4501126d1" member_type: VOTER last_known_addr { host: "127.16.217.60" port: 39253 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:21:36.851950 7326 raft_consensus.cc:348] T 00000000000000000000000000000000 P b8e62a119beb46cea5763e672f95303a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f180af9b6d7043d7a7ba302ffedeeb3a" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 42485 } } peers { permanent_uuid: "b8e62a119beb46cea5763e672f95303a" member_type: VOTER last_known_addr { host: "127.16.217.61" port: 36701 } } peers { permanent_uuid: "1513f03de35d4f828fa4cfe4501126d1" member_type: VOTER last_known_addr { host: "127.16.217.60" port: 39253 } }
I20260620 20:21:36.852401 7326 raft_consensus.cc:729] T 00000000000000000000000000000000 P b8e62a119beb46cea5763e672f95303a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8e62a119beb46cea5763e672f95303a, State: Initialized, Role: FOLLOWER
I20260620 20:21:36.852948 7326 consensus_queue.cc:260] T 00000000000000000000000000000000 P b8e62a119beb46cea5763e672f95303a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f180af9b6d7043d7a7ba302ffedeeb3a" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 42485 } } peers { permanent_uuid: "b8e62a119beb46cea5763e672f95303a" member_type: VOTER last_known_addr { host: "127.16.217.61" port: 36701 } } peers { permanent_uuid: "1513f03de35d4f828fa4cfe4501126d1" member_type: VOTER last_known_addr { host: "127.16.217.60" port: 39253 } }
I20260620 20:21:36.872506 7326 sys_catalog.cc:564] T 00000000000000000000000000000000 P b8e62a119beb46cea5763e672f95303a [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:21:36.872556 7301 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f180af9b6d7043d7a7ba302ffedeeb3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "b8e62a119beb46cea5763e672f95303a"
I20260620 20:21:36.873507 7300 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b8e62a119beb46cea5763e672f95303a [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:21:36.878654 7300 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b8e62a119beb46cea5763e672f95303a [term 1 FOLLOWER]: Refusing update from remote peer f180af9b6d7043d7a7ba302ffedeeb3a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:21:36.879926 7431 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f180af9b6d7043d7a7ba302ffedeeb3a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8e62a119beb46cea5763e672f95303a" member_type: VOTER last_known_addr { host: "127.16.217.61" port: 36701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:21:36.882329 7301 raft_consensus.cc:2454] T 00000000000000000000000000000000 P b8e62a119beb46cea5763e672f95303a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f180af9b6d7043d7a7ba302ffedeeb3a in term 1.
I20260620 20:21:36.882540 7447 sys_catalog.cc:455] T 00000000000000000000000000000000 P b8e62a119beb46cea5763e672f95303a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f180af9b6d7043d7a7ba302ffedeeb3a" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 42485 } } peers { permanent_uuid: "b8e62a119beb46cea5763e672f95303a" member_type: VOTER last_known_addr { host: "127.16.217.61" port: 36701 } } peers { permanent_uuid: "1513f03de35d4f828fa4cfe4501126d1" member_type: VOTER last_known_addr { host: "127.16.217.60" port: 39253 } } }
I20260620 20:21:36.883026 7447 sys_catalog.cc:458] T 00000000000000000000000000000000 P b8e62a119beb46cea5763e672f95303a [sys.catalog]: This master's current role is: FOLLOWER
I20260620 20:21:36.891425 7406 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1513f03de35d4f828fa4cfe4501126d1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f180af9b6d7043d7a7ba302ffedeeb3a. Latest consensus state: current_term: 1 leader_uuid: "f180af9b6d7043d7a7ba302ffedeeb3a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f180af9b6d7043d7a7ba302ffedeeb3a" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 42485 } } peers { permanent_uuid: "b8e62a119beb46cea5763e672f95303a" member_type: VOTER last_known_addr { host: "127.16.217.61" port: 36701 } } peers { permanent_uuid: "1513f03de35d4f828fa4cfe4501126d1" member_type: VOTER last_known_addr { host: "127.16.217.60" port: 39253 } } }
I20260620 20:21:36.891752 7406 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1513f03de35d4f828fa4cfe4501126d1 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 20:21:36.895617 7446 mvcc.cc:205] Tried to move back new op lower bound from 7299018329454493696 to 7299018329034305536. Current Snapshot: MvccSnapshot[applied={T|T < 7299018329454493696}]
I20260620 20:21:36.895576 7431 sys_catalog.cc:455] T 00000000000000000000000000000000 P f180af9b6d7043d7a7ba302ffedeeb3a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f180af9b6d7043d7a7ba302ffedeeb3a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f180af9b6d7043d7a7ba302ffedeeb3a" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 42485 } } peers { permanent_uuid: "b8e62a119beb46cea5763e672f95303a" member_type: VOTER last_known_addr { host: "127.16.217.61" port: 36701 } } peers { permanent_uuid: "1513f03de35d4f828fa4cfe4501126d1" member_type: VOTER last_known_addr { host: "127.16.217.60" port: 39253 } } }
I20260620 20:21:36.895840 7431 sys_catalog.cc:458] T 00000000000000000000000000000000 P f180af9b6d7043d7a7ba302ffedeeb3a [sys.catalog]: This master's current role is: LEADER
I20260620 20:21:36.902637 7437 catalog_manager.cc:1347] Generated new cluster ID: 60db223d04c94f85aefc88d692d76926
I20260620 20:21:36.902772 7437 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 20:21:36.911692 7406 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1513f03de35d4f828fa4cfe4501126d1 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f180af9b6d7043d7a7ba302ffedeeb3a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f180af9b6d7043d7a7ba302ffedeeb3a" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 42485 } } peers { permanent_uuid: "b8e62a119beb46cea5763e672f95303a" member_type: VOTER last_known_addr { host: "127.16.217.61" port: 36701 } } peers { permanent_uuid: "1513f03de35d4f828fa4cfe4501126d1" member_type: VOTER last_known_addr { host: "127.16.217.60" port: 39253 } } }
I20260620 20:21:36.911998 7406 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1513f03de35d4f828fa4cfe4501126d1 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 20:21:36.922864 7447 sys_catalog.cc:455] T 00000000000000000000000000000000 P b8e62a119beb46cea5763e672f95303a [sys.catalog]: SysCatalogTable state changed. Reason: New leader f180af9b6d7043d7a7ba302ffedeeb3a. Latest consensus state: current_term: 1 leader_uuid: "f180af9b6d7043d7a7ba302ffedeeb3a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f180af9b6d7043d7a7ba302ffedeeb3a" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 42485 } } peers { permanent_uuid: "b8e62a119beb46cea5763e672f95303a" member_type: VOTER last_known_addr { host: "127.16.217.61" port: 36701 } } peers { permanent_uuid: "1513f03de35d4f828fa4cfe4501126d1" member_type: VOTER last_known_addr { host: "127.16.217.60" port: 39253 } } }
I20260620 20:21:36.923246 7447 sys_catalog.cc:458] T 00000000000000000000000000000000 P b8e62a119beb46cea5763e672f95303a [sys.catalog]: This master's current role is: FOLLOWER
I20260620 20:21:36.925508 7431 sys_catalog.cc:455] T 00000000000000000000000000000000 P f180af9b6d7043d7a7ba302ffedeeb3a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f180af9b6d7043d7a7ba302ffedeeb3a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f180af9b6d7043d7a7ba302ffedeeb3a" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 42485 } } peers { permanent_uuid: "b8e62a119beb46cea5763e672f95303a" member_type: VOTER last_known_addr { host: "127.16.217.61" port: 36701 } } peers { permanent_uuid: "1513f03de35d4f828fa4cfe4501126d1" member_type: VOTER last_known_addr { host: "127.16.217.60" port: 39253 } } }
I20260620 20:21:36.925802 7431 sys_catalog.cc:458] T 00000000000000000000000000000000 P f180af9b6d7043d7a7ba302ffedeeb3a [sys.catalog]: This master's current role is: LEADER
I20260620 20:21:36.938462 7447 sys_catalog.cc:455] T 00000000000000000000000000000000 P b8e62a119beb46cea5763e672f95303a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f180af9b6d7043d7a7ba302ffedeeb3a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f180af9b6d7043d7a7ba302ffedeeb3a" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 42485 } } peers { permanent_uuid: "b8e62a119beb46cea5763e672f95303a" member_type: VOTER last_known_addr { host: "127.16.217.61" port: 36701 } } peers { permanent_uuid: "1513f03de35d4f828fa4cfe4501126d1" member_type: VOTER last_known_addr { host: "127.16.217.60" port: 39253 } } }
I20260620 20:21:36.938750 7447 sys_catalog.cc:458] T 00000000000000000000000000000000 P b8e62a119beb46cea5763e672f95303a [sys.catalog]: This master's current role is: FOLLOWER
I20260620 20:21:36.959280 7437 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 20:21:36.961350 7437 catalog_manager.cc:1504] Loading token signing keys...
I20260620 20:21:36.992128 7437 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f180af9b6d7043d7a7ba302ffedeeb3a: Generated new TSK 0
I20260620 20:21:36.993263 7437 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 20:21:37.019776 7465 catalog_manager.cc:1259] Loaded cluster ID: 60db223d04c94f85aefc88d692d76926
I20260620 20:21:37.019994 7465 catalog_manager.cc:1552] T 00000000000000000000000000000000 P b8e62a119beb46cea5763e672f95303a: loading cluster ID for follower catalog manager: success
I20260620 20:21:37.023927 7465 catalog_manager.cc:1574] T 00000000000000000000000000000000 P b8e62a119beb46cea5763e672f95303a: acquiring CA information for follower catalog manager: success
I20260620 20:21:37.026552 7465 catalog_manager.cc:1602] T 00000000000000000000000000000000 P b8e62a119beb46cea5763e672f95303a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260620 20:21:37.136590 7256 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:21:37.128402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48858 (local address 127.16.217.62:42485)
0620 20:21:37.128665 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0620 20:21:37.128673 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 20:21:37.134490 (+ 5817us) server_negotiation.cc:407] Connection header received
0620 20:21:37.134817 (+ 327us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:21:37.134825 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:21:37.134918 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:21:37.135023 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:21:37.136310 (+ 1287us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48858: BlockingRecv error: recv got EOF from 127.0.0.1:48858 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
W20260620 20:21:37.139127 7403 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:21:37.134535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37406 (local address 127.16.217.60:39253)
0620 20:21:37.134709 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0620 20:21:37.134715 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:21:37.137413 (+ 2698us) server_negotiation.cc:407] Connection header received
0620 20:21:37.137670 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:21:37.137679 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:21:37.137763 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:21:37.137840 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:21:37.138869 (+ 1029us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37406: BlockingRecv error: recv got EOF from 127.0.0.1:37406 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
W20260620 20:21:37.144001 7330 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:21:37.135531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46164 (local address 127.16.217.61:36701)
0620 20:21:37.135766 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0620 20:21:37.135775 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 20:21:37.142290 (+ 6515us) server_negotiation.cc:407] Connection header received
0620 20:21:37.142556 (+ 266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:21:37.142563 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:21:37.142656 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:21:37.142736 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:21:37.143728 (+ 992us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46164: BlockingRecv error: recv got EOF from 127.0.0.1:46164 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 20:21:37.250155 7330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:37.223827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46180 (local address 127.16.217.61:36701)
0620 20:21:37.224091 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0620 20:21:37.224103 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 20:21:37.242505 (+ 18402us) server_negotiation.cc:407] Connection header received
0620 20:21:37.242794 (+ 289us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:21:37.242801 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:21:37.243043 (+ 242us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:21:37.243201 (+ 158us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 20:21:37.243906 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:37.245079 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:37.247117 (+ 2038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:37.247783 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:37.247885 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:21:37.248619 (+ 734us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 20:21:37.249320 (+ 701us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 20:21:37.249436 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0620 20:21:37.249681 (+ 245us) server_negotiation.cc:299] Negotiation successful
0620 20:21:37.249785 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
W20260620 20:21:37.316653 7403 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:21:37.304237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37422 (local address 127.16.217.60:39253)
0620 20:21:37.304441 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0620 20:21:37.304447 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:21:37.314690 (+ 10243us) server_negotiation.cc:407] Connection header received
0620 20:21:37.314887 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:21:37.314894 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:21:37.315047 (+ 153us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:21:37.315155 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:21:37.316389 (+ 1234us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37422: BlockingRecv error: recv got EOF from 127.0.0.1:37422 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 20:21:37.374867 7330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:37.367869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46186 (local address 127.16.217.61:36701)
0620 20:21:37.368096 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0620 20:21:37.368103 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:21:37.368293 (+ 190us) server_negotiation.cc:407] Connection header received
0620 20:21:37.368621 (+ 328us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:21:37.368628 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:21:37.368714 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:21:37.368822 (+ 108us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 20:21:37.369424 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:37.370549 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:37.372615 (+ 2066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:37.373280 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:37.373431 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:21:37.374041 (+ 610us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 20:21:37.374198 (+ 157us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 20:21:37.374286 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0620 20:21:37.374476 (+ 190us) server_negotiation.cc:299] Negotiation successful
0620 20:21:37.374558 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
W20260620 20:21:37.388022 7403 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:21:37.383246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37438 (local address 127.16.217.60:39253)
0620 20:21:37.383471 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0620 20:21:37.383479 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 20:21:37.386475 (+ 2996us) server_negotiation.cc:407] Connection header received
0620 20:21:37.386694 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:21:37.386701 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:21:37.386801 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:21:37.386906 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:21:37.387787 (+ 881us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37438: BlockingRecv error: recv got EOF from 127.0.0.1:37438 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
I20260620 20:21:37.407332 7330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:37.399863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46202 (local address 127.16.217.61:36701)
0620 20:21:37.400055 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0620 20:21:37.400062 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:21:37.400583 (+ 521us) server_negotiation.cc:407] Connection header received
0620 20:21:37.400790 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:21:37.400796 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:21:37.400909 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:21:37.401051 (+ 142us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 20:21:37.401569 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:37.402798 (+ 1229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:37.404835 (+ 2037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:37.405560 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:37.405824 (+ 264us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:21:37.406353 (+ 529us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 20:21:37.406565 (+ 212us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 20:21:37.406673 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0620 20:21:37.406831 (+ 158us) server_negotiation.cc:299] Negotiation successful
0620 20:21:37.406940 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
W20260620 20:21:37.422222 7403 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:21:37.419801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37454 (local address 127.16.217.60:39253)
0620 20:21:37.420002 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0620 20:21:37.420011 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 20:21:37.420588 (+ 577us) server_negotiation.cc:407] Connection header received
0620 20:21:37.420781 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:21:37.420790 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:21:37.420901 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:21:37.421039 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:21:37.421956 (+ 917us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37454: BlockingRecv error: recv got EOF from 127.0.0.1:37454 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 20:21:37.437117 7330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:37.429754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46206 (local address 127.16.217.61:36701)
0620 20:21:37.430023 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0620 20:21:37.430029 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:21:37.430532 (+ 503us) server_negotiation.cc:407] Connection header received
0620 20:21:37.430761 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:21:37.430769 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:21:37.430882 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:21:37.431025 (+ 143us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 20:21:37.431522 (+ 497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:37.432841 (+ 1319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:37.434804 (+ 1963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:37.435593 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:37.435743 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:21:37.436240 (+ 497us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 20:21:37.436396 (+ 156us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 20:21:37.436518 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0620 20:21:37.436664 (+ 146us) server_negotiation.cc:299] Negotiation successful
0620 20:21:37.436759 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136}
W20260620 20:21:37.452180 7403 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:21:37.444575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37462 (local address 127.16.217.60:39253)
0620 20:21:37.444859 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0620 20:21:37.444866 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:21:37.450461 (+ 5595us) server_negotiation.cc:407] Connection header received
0620 20:21:37.450773 (+ 312us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:21:37.450780 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:21:37.450879 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:21:37.451006 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:21:37.451919 (+ 913us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37462: BlockingRecv error: recv got EOF from 127.0.0.1:37462 (error 108)
Metrics: {"server-negotiator.queue_time_us":152}
I20260620 20:21:37.475961 7330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:37.468131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46214 (local address 127.16.217.61:36701)
0620 20:21:37.468344 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0620 20:21:37.468352 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 20:21:37.468996 (+ 644us) server_negotiation.cc:407] Connection header received
0620 20:21:37.469242 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:21:37.469248 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:21:37.469338 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:21:37.469458 (+ 120us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 20:21:37.470098 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:37.471301 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:37.473449 (+ 2148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:37.474282 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:37.474423 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:21:37.475025 (+ 602us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 20:21:37.475164 (+ 139us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 20:21:37.475259 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0620 20:21:37.475501 (+ 242us) server_negotiation.cc:299] Negotiation successful
0620 20:21:37.475590 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
W20260620 20:21:37.486713 7403 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:21:37.483963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37476 (local address 127.16.217.60:39253)
0620 20:21:37.484187 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0620 20:21:37.484196 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 20:21:37.484755 (+ 559us) server_negotiation.cc:407] Connection header received
0620 20:21:37.484966 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:21:37.484972 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:21:37.485086 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:21:37.485201 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:21:37.486453 (+ 1252us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37476: BlockingRecv error: recv got EOF from 127.0.0.1:37476 (error 108)
Metrics: {"server-negotiator.queue_time_us":86}
I20260620 20:21:37.514088 7330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:37.506769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46220 (local address 127.16.217.61:36701)
0620 20:21:37.506973 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0620 20:21:37.506981 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 20:21:37.507530 (+ 549us) server_negotiation.cc:407] Connection header received
0620 20:21:37.507727 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:21:37.507734 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:21:37.507837 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:21:37.507964 (+ 127us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 20:21:37.508473 (+ 509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:37.509674 (+ 1201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:37.511690 (+ 2016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:37.512393 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:37.512540 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:21:37.513151 (+ 611us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 20:21:37.513308 (+ 157us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 20:21:37.513413 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0620 20:21:37.513602 (+ 189us) server_negotiation.cc:299] Negotiation successful
0620 20:21:37.513696 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
W20260620 20:21:37.531749 7403 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:21:37.528755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37492 (local address 127.16.217.60:39253)
0620 20:21:37.529085 (+ 330us) server_negotiation.cc:206] Beginning negotiation
0620 20:21:37.529092 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:21:37.530130 (+ 1038us) server_negotiation.cc:407] Connection header received
0620 20:21:37.530327 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:21:37.530333 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:21:37.530449 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:21:37.530564 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:21:37.531483 (+ 919us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37492: BlockingRecv error: recv got EOF from 127.0.0.1:37492 (error 108)
Metrics: {"server-negotiator.queue_time_us":172}
I20260620 20:21:37.586737 7330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:37.580174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46226 (local address 127.16.217.61:36701)
0620 20:21:37.580524 (+ 350us) server_negotiation.cc:206] Beginning negotiation
0620 20:21:37.580535 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 20:21:37.581164 (+ 629us) server_negotiation.cc:407] Connection header received
0620 20:21:37.581477 (+ 313us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:21:37.581486 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:21:37.581587 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:21:37.581679 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 20:21:37.582524 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:37.583301 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:37.584702 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:37.585208 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:37.585363 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:21:37.585799 (+ 436us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 20:21:37.585966 (+ 167us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 20:21:37.586074 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0620 20:21:37.586241 (+ 167us) server_negotiation.cc:299] Negotiation successful
0620 20:21:37.586327 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
W20260620 20:21:37.595046 7403 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:21:37.593123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37500 (local address 127.16.217.60:39253)
0620 20:21:37.593320 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0620 20:21:37.593332 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 20:21:37.593468 (+ 136us) server_negotiation.cc:407] Connection header received
0620 20:21:37.593752 (+ 284us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:21:37.593759 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:21:37.593934 (+ 175us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:21:37.594068 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:21:37.594835 (+ 767us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37500: BlockingRecv error: recv got EOF from 127.0.0.1:37500 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
I20260620 20:21:37.663524 7417 catalog_manager.cc:1259] Loaded cluster ID: 60db223d04c94f85aefc88d692d76926
I20260620 20:21:37.663619 7417 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 1513f03de35d4f828fa4cfe4501126d1: loading cluster ID for follower catalog manager: success
I20260620 20:21:37.666783 7417 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 1513f03de35d4f828fa4cfe4501126d1: acquiring CA information for follower catalog manager: success
I20260620 20:21:37.669361 7417 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 1513f03de35d4f828fa4cfe4501126d1: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 20:21:37.670707 7330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:37.664529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46234 (local address 127.16.217.61:36701)
0620 20:21:37.664735 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0620 20:21:37.664741 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:21:37.665033 (+ 292us) server_negotiation.cc:407] Connection header received
0620 20:21:37.665238 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:21:37.665241 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:21:37.665309 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:21:37.665387 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 20:21:37.665948 (+ 561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:37.666746 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:37.668941 (+ 2195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:37.669359 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:37.669454 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:21:37.669983 (+ 529us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 20:21:37.670092 (+ 109us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 20:21:37.670165 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 20:21:37.670325 (+ 160us) server_negotiation.cc:299] Negotiation successful
0620 20:21:37.670388 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260620 20:21:37.681388 7403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:37.676300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37508 (local address 127.16.217.60:39253)
0620 20:21:37.676579 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0620 20:21:37.676585 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:21:37.676835 (+ 250us) server_negotiation.cc:407] Connection header received
0620 20:21:37.676982 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:21:37.676986 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:21:37.677103 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:21:37.677206 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 20:21:37.677534 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:37.678264 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:37.679499 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:37.679870 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:37.679966 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:21:37.680355 (+ 389us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 20:21:37.680814 (+ 459us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 20:21:37.680920 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0620 20:21:37.681024 (+ 104us) server_negotiation.cc:299] Negotiation successful
0620 20:21:37.681082 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260620 20:21:37.695393 7403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:37.687828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37510 (local address 127.16.217.60:39253)
0620 20:21:37.688006 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0620 20:21:37.688012 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:21:37.688910 (+ 898us) server_negotiation.cc:407] Connection header received
0620 20:21:37.689202 (+ 292us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:21:37.689210 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:21:37.689302 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:21:37.689370 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 20:21:37.690450 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:37.691144 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:37.693152 (+ 2008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:37.693843 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:37.694014 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:21:37.694453 (+ 439us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 20:21:37.694594 (+ 141us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 20:21:37.694689 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0620 20:21:37.694981 (+ 292us) server_negotiation.cc:299] Negotiation successful
0620 20:21:37.695062 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 20:21:37.696049 7330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:37.687841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46238 (local address 127.16.217.61:36701)
0620 20:21:37.688040 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0620 20:21:37.688049 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 20:21:37.689142 (+ 1093us) server_negotiation.cc:407] Connection header received
0620 20:21:37.689376 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:21:37.689381 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:21:37.689456 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:21:37.689556 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 20:21:37.690517 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:37.691637 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:37.693800 (+ 2163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:37.694460 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:37.694594 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:21:37.695245 (+ 651us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 20:21:37.695388 (+ 143us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 20:21:37.695450 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0620 20:21:37.695673 (+ 223us) server_negotiation.cc:299] Negotiation successful
0620 20:21:37.695753 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 20:21:37.696734 7562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:37.687134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48862 (local address 127.16.217.62:42485)
0620 20:21:37.689253 (+ 2119us) server_negotiation.cc:206] Beginning negotiation
0620 20:21:37.689258 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:21:37.689273 (+ 15us) server_negotiation.cc:407] Connection header received
0620 20:21:37.689397 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:21:37.689402 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:21:37.689602 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:21:37.689767 (+ 165us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 20:21:37.690519 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:37.691675 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:37.693620 (+ 1945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:37.694287 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:37.694421 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:21:37.695405 (+ 984us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 20:21:37.695846 (+ 441us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 20:21:37.695945 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0620 20:21:37.696118 (+ 173us) server_negotiation.cc:299] Negotiation successful
0620 20:21:37.696224 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2015,"thread_start_us":1927,"threads_started":1}
I20260620 20:21:37.702315 17252 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu pid 7185
I20260620 20:21:37.703348 7247 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:21:37.811487 17252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu with pid 7185
I20260620 20:21:37.832098 17252 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu pid 7259
I20260620 20:21:37.833205 7321 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:21:37.938832 17252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu with pid 7259
I20260620 20:21:37.959448 17252 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu pid 7333
I20260620 20:21:37.960673 7395 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:21:38.067490 17252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu with pid 7333
2026-06-20T20:21:38Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (4331 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986536687152-17252-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 20:21:38 dist-test-slave-2hpg krb5kdc[7570](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 20:21:38 dist-test-slave-2hpg krb5kdc[7570](info): set up 2 sockets
Jun 20 20:21:38 dist-test-slave-2hpg krb5kdc[7570](info): commencing operation
krb5kdc: starting...
W20260620 20:21:39.186919 17252 mini_kdc.cc:121] Time spent starting KDC: real 1.079s user 0.002s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986536687152-17252-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986536687152-17252-0/krb5kdc/test-user.keytab.
Jun 20 20:21:39 dist-test-slave-2hpg krb5kdc[7570](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986899, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T20:21:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T20:21:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.217.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.217.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.217.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.62.keytab.
Entry for principal kudu/127.16.217.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.62.keytab.
I20260620 20:21:39.457309 17252 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
/tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986536687152-17252-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986536687152-17252-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986536687152-17252-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986536687152-17252-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.217.62:43175
--webserver_interface=127.16.217.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.62.keytab
--principal=kudu/127.16.217.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.217.20:44807
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.217.62:43175,127.16.217.61:33651,127.16.217.60:34865
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986536687152-17252-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986536687152-17252-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986536687152-17252-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:21:39.789134 7583 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 20:21:39.789446 7583 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:21:39.789521 7583 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:21:39.789608 7583 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:21:39.796909 7583 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:21:39.797001 7583 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:21:39.797063 7583 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:21:39.797114 7583 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:21:39.797176 7583 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:21:39.807050 7583 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.217.20:44807
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986536687152-17252-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986536687152-17252-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.217.62:43175,127.16.217.61:33651,127.16.217.60:34865
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.62.keytab
--principal=kudu/127.16.217.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.217.62:43175
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986536687152-17252-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.217.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986536687152-17252-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:21:39.808274 7583 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:21:39.809545 7583 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:21:39.821913 7589 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:21:39.822041 7591 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:21:39.821810 7588 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:21:39.822934 7583 server_base.cc:1034] running on GCE node
I20260620 20:21:39.823818 7583 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:21:39.825325 7583 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:21:39.826649 7583 hybrid_clock.cc:648] HybridClock initialized: now 1781986899826584 us; error 66 us; skew 500 ppm
I20260620 20:21:39.827010 7583 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:21:39 dist-test-slave-2hpg krb5kdc[7570](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986899, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.217.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:21:39.830161 7583 init.cc:377] Logged in from keytab as kudu/127.16.217.62@KRBTEST.COM (short username kudu)
I20260620 20:21:39.831566 7583 webserver.cc:460] Webserver started at http://127.16.217.62:36901/ using document root <none> and password file <none>
I20260620 20:21:39.832188 7583 fs_manager.cc:359] Metadata directory not provided
I20260620 20:21:39.832273 7583 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:21:39.832537 7583 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:21:39.835402 7583 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986536687152-17252-0/minicluster-data/master-0/data/instance:
uuid: "fb0dd4cdadea49458ae5e99c8f666997"
format_stamp: "Formatted at 2026-06-20 20:21:39 on dist-test-slave-2hpg"
server_key: "29d14300e9226a2b73e39d52310cea6f"
server_key_iv: "1933548dbab36bb76e9a0a3fde874b4f"
server_key_version: "encryptionkey@0"
I20260620 20:21:39.836139 7583 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986536687152-17252-0/minicluster-data/master-0/wal/instance:
uuid: "fb0dd4cdadea49458ae5e99c8f666997"
format_stamp: "Formatted at 2026-06-20 20:21:39 on dist-test-slave-2hpg"
server_key: "29d14300e9226a2b73e39d52310cea6f"
server_key_iv: "1933548dbab36bb76e9a0a3fde874b4f"
server_key_version: "encryptionkey@0"
I20260620 20:21:39.841127 7583 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260620 20:21:39.844609 7598 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:21:39.846206 7583 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 20:21:39.846390 7583 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986536687152-17252-0/minicluster-data/master-0/data,/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986536687152-17252-0/minicluster-data/master-0/wal
uuid: "fb0dd4cdadea49458ae5e99c8f666997"
format_stamp: "Formatted at 2026-06-20 20:21:39 on dist-test-slave-2hpg"
server_key: "29d14300e9226a2b73e39d52310cea6f"
server_key_iv: "1933548dbab36bb76e9a0a3fde874b4f"
server_key_version: "encryptionkey@0"
I20260620 20:21:39.846606 7583 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986536687152-17252-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986536687152-17252-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986536687152-17252-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:21:39.861546 7583 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:21:39.865496 7583 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:21:39.865772 7583 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:21:39.884030 7583 rpc_server.cc:307] RPC server started. Bound to: 127.16.217.62:43175
I20260620 20:21:39.884052 7649 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.217.62:43175 every 8 connection(s)
I20260620 20:21:39.885377 7583 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986536687152-17252-0/minicluster-data/master-0/data/info.pb
I20260620 20:21:39.889024 7650 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:21:39.892005 17252 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu as pid 7583
I20260620 20:21:39.892203 17252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986536687152-17252-0/minicluster-data/master-0/wal/instance
I20260620 20:21:39.892654 17252 external_mini_cluster.cc:1428] Setting key 03fb692ac308400159c9b7781b26c045
I20260620 20:21:39.895574 7650 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.217.62" port: 43175 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 20:21:39 dist-test-slave-2hpg krb5kdc[7570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986899, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.217.62@KRBTEST.COM for kudu/127.16.217.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.16.217.61@KRBTEST.COM; defaulting to no policy
I20260620 20:21:39.919826 7653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:39.899733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58608 (local address 127.16.217.62:43175)
0620 20:21:39.900485 (+ 752us) server_negotiation.cc:206] Beginning negotiation
0620 20:21:39.900496 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 20:21:39.900554 (+ 58us) server_negotiation.cc:407] Connection header received
0620 20:21:39.902124 (+ 1570us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:21:39.902131 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:21:39.902402 (+ 271us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:21:39.902597 (+ 195us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:21:39.904237 (+ 1640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:39.905506 (+ 1269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:39.907744 (+ 2238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:39.908328 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:39.908457 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:21:39.912500 (+ 4043us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:21:39.912508 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:21:39.912520 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:21:39.912551 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:21:39.915471 (+ 2920us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:21:39.916184 (+ 713us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:21:39.916190 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:21:39.916193 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:21:39.916337 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:21:39.916727 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:21:39.916734 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:21:39.916741 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:21:39.917263 (+ 522us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:21:39.917471 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0620 20:21:39.918019 (+ 548us) server_negotiation.cc:299] Negotiation successful
0620 20:21:39.918344 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":509,"thread_start_us":333,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.217.61@KRBTEST.COM" created.
I20260620 20:21:39.919839 7652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:39.898500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:43175 (local address 127.0.0.1:58608)
0620 20:21:39.899303 (+ 803us) negotiation.cc:107] Waiting for socket to connect
0620 20:21:39.899348 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 20:21:39.901680 (+ 2332us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:21:39.902661 (+ 981us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:21:39.902671 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:21:39.903480 (+ 809us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:21:39.903995 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:21:39.904023 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:21:39.905670 (+ 1647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:21:39.905678 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:21:39.907602 (+ 1924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:21:39.907620 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:21:39.908558 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:21:39.908567 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:21:39.908815 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:21:39.909522 (+ 707us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:21:39.909563 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:21:39.912262 (+ 2699us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:21:39.915623 (+ 3361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:21:39.915636 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:21:39.915648 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:21:39.916035 (+ 387us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:21:39.916446 (+ 411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:21:39.916452 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:21:39.916455 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:21:39.916616 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:21:39.917402 (+ 786us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:21:39.917410 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:21:39.917620 (+ 210us) client_negotiation.cc:769] Sending connection context
0620 20:21:39.917864 (+ 244us) client_negotiation.cc:240] Negotiation successful
0620 20:21:39.918462 (+ 598us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":541,"mutex_wait_us":136,"thread_start_us":345,"threads_started":1}
I20260620 20:21:39.923112 7650 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.217.61" port: 33651 } has no permanent_uuid. Determining permanent_uuid...
I20260620 20:21:39.924258 7652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:39.923835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.61:33651 (local address 127.0.0.1:58374)
0620 20:21:39.924035 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0620 20:21:39.924146 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.217.61:33651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260620 20:21:39.924551 7600 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.61:33651: connect: Connection refused (error 111)
I20260620 20:21:39.926162 7652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:39.925775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.61:33651 (local address 127.0.0.1:58390)
0620 20:21:39.925948 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0620 20:21:39.926031 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.217.61:33651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260620 20:21:39.926373 7650 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.217.61:33651: Network error: Client connection negotiation failed: client connection to 127.16.217.61:33651: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.217.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.61.keytab.
Entry for principal kudu/127.16.217.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.61.keytab.
I20260620 20:21:39.953094 17252 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
/tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986536687152-17252-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986536687152-17252-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986536687152-17252-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986536687152-17252-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.217.61:33651
--webserver_interface=127.16.217.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.61.keytab
--principal=kudu/127.16.217.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.217.20:44807
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.217.62:43175,127.16.217.61:33651,127.16.217.60:34865
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986536687152-17252-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986536687152-17252-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986536687152-17252-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 20:21:39.975731 7650 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.217.61" port: 33651 } attempt: 1
I20260620 20:21:39.977092 7652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:39.976555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.61:33651 (local address 127.0.0.1:58400)
0620 20:21:39.976783 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0620 20:21:39.976923 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.217.61:33651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260620 20:21:39.978237 7652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:39.977815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.61:33651 (local address 127.0.0.1:58414)
0620 20:21:39.977996 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0620 20:21:39.978078 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.217.61:33651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
W20260620 20:21:39.978490 7650 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.217.61:33651: Network error: Client connection negotiation failed: client connection to 127.16.217.61:33651: connect: Connection refused (error 111)
I20260620 20:21:40.046751 7650 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.217.61" port: 33651 } attempt: 2
I20260620 20:21:40.048075 7652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:40.047423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.61:33651 (local address 127.0.0.1:58418)
0620 20:21:40.047716 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0620 20:21:40.047860 (+ 144us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.217.61:33651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260620 20:21:40.049230 7652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:40.048811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.61:33651 (local address 127.0.0.1:58422)
0620 20:21:40.048983 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0620 20:21:40.049068 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.217.61:33651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260620 20:21:40.049424 7650 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.217.61:33651: Network error: Client connection negotiation failed: client connection to 127.16.217.61:33651: connect: Connection refused (error 111)
I20260620 20:21:40.140692 7650 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.217.61" port: 33651 } attempt: 3
I20260620 20:21:40.141963 7652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:40.141424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.61:33651 (local address 127.0.0.1:58428)
0620 20:21:40.141649 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0620 20:21:40.141760 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.217.61:33651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260620 20:21:40.143137 7652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:40.142646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.61:33651 (local address 127.0.0.1:58444)
0620 20:21:40.142830 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0620 20:21:40.142938 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.217.61:33651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260620 20:21:40.143388 7650 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.217.61:33651: Network error: Client connection negotiation failed: client connection to 127.16.217.61:33651: connect: Connection refused (error 111)
I20260620 20:21:40.286657 7650 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.217.61" port: 33651 } attempt: 4
W20260620 20:21:40.286513 7657 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 20:21:40.286949 7657 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:21:40.287104 7657 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:21:40.287230 7657 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260620 20:21:40.288017 7652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:40.287414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.61:33651 (local address 127.0.0.1:58452)
0620 20:21:40.287751 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0620 20:21:40.287853 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.217.61:33651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179}
I20260620 20:21:40.289091 7652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:40.288704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.61:33651 (local address 127.0.0.1:58468)
0620 20:21:40.288853 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0620 20:21:40.288953 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.217.61:33651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260620 20:21:40.289376 7650 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.217.61:33651: Network error: Client connection negotiation failed: client connection to 127.16.217.61:33651: connect: Connection refused (error 111)
W20260620 20:21:40.295300 7657 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:21:40.295387 7657 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:21:40.295445 7657 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:21:40.295545 7657 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:21:40.295637 7657 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:21:40.305537 7657 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.217.20:44807
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986536687152-17252-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986536687152-17252-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.16.217.62:43175,127.16.217.61:33651,127.16.217.60:34865
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.61.keytab
--principal=kudu/127.16.217.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.217.61:33651
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986536687152-17252-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.16.217.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986536687152-17252-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:21:40.306900 7657 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:21:40.308261 7657 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:21:40.320819 7663 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:21:40.320847 7665 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:21:40.321044 7662 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:21:40.322109 7657 server_base.cc:1034] running on GCE node
I20260620 20:21:40.322937 7657 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:21:40.324431 7657 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:21:40.325711 7657 hybrid_clock.cc:648] HybridClock initialized: now 1781986900325651 us; error 36 us; skew 500 ppm
I20260620 20:21:40.326136 7657 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:21:40 dist-test-slave-2hpg krb5kdc[7570](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986900, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.217.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:21:40.329527 7657 init.cc:377] Logged in from keytab as kudu/127.16.217.61@KRBTEST.COM (short username kudu)
I20260620 20:21:40.331177 7657 webserver.cc:460] Webserver started at http://127.16.217.61:40269/ using document root <none> and password file <none>
I20260620 20:21:40.331900 7657 fs_manager.cc:359] Metadata directory not provided
I20260620 20:21:40.332041 7657 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:21:40.332366 7657 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:21:40.335457 7657 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986536687152-17252-0/minicluster-data/master-1/data/instance:
uuid: "38c990bb69be4a4488cce8c5bbccb219"
format_stamp: "Formatted at 2026-06-20 20:21:40 on dist-test-slave-2hpg"
server_key: "d71cc02e93e4cdc64918b7d948029242"
server_key_iv: "119db64676bc1106ce611fb1d8a4765b"
server_key_version: "encryptionkey@0"
I20260620 20:21:40.336274 7657 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986536687152-17252-0/minicluster-data/master-1/wal/instance:
uuid: "38c990bb69be4a4488cce8c5bbccb219"
format_stamp: "Formatted at 2026-06-20 20:21:40 on dist-test-slave-2hpg"
server_key: "d71cc02e93e4cdc64918b7d948029242"
server_key_iv: "119db64676bc1106ce611fb1d8a4765b"
server_key_version: "encryptionkey@0"
I20260620 20:21:40.341678 7657 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260620 20:21:40.345654 7672 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:21:40.347136 7657 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 20:21:40.347281 7657 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986536687152-17252-0/minicluster-data/master-1/data,/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986536687152-17252-0/minicluster-data/master-1/wal
uuid: "38c990bb69be4a4488cce8c5bbccb219"
format_stamp: "Formatted at 2026-06-20 20:21:40 on dist-test-slave-2hpg"
server_key: "d71cc02e93e4cdc64918b7d948029242"
server_key_iv: "119db64676bc1106ce611fb1d8a4765b"
server_key_version: "encryptionkey@0"
I20260620 20:21:40.347471 7657 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986536687152-17252-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986536687152-17252-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986536687152-17252-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:21:40.376509 7657 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:21:40.380411 7657 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:21:40.380640 7657 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:21:40.399497 7657 rpc_server.cc:307] RPC server started. Bound to: 127.16.217.61:33651
I20260620 20:21:40.399506 7723 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.217.61:33651 every 8 connection(s)
I20260620 20:21:40.400828 7657 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986536687152-17252-0/minicluster-data/master-1/data/info.pb
I20260620 20:21:40.404564 7724 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:21:40.407871 17252 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu as pid 7657
I20260620 20:21:40.408097 17252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986536687152-17252-0/minicluster-data/master-1/wal/instance
I20260620 20:21:40.408617 17252 external_mini_cluster.cc:1428] Setting key fd36ea04b9cee7ec63329df36228b868
I20260620 20:21:40.413347 7724 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.217.62" port: 43175 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 20:21:40 dist-test-slave-2hpg krb5kdc[7570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986900, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.217.61@KRBTEST.COM for kudu/127.16.217.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.16.217.60@KRBTEST.COM; defaulting to no policy
I20260620 20:21:40.436547 7653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:40.416868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48246 (local address 127.16.217.62:43175)
0620 20:21:40.417110 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0620 20:21:40.417119 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 20:21:40.418297 (+ 1178us) server_negotiation.cc:407] Connection header received
0620 20:21:40.420356 (+ 2059us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:21:40.420360 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:21:40.420433 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:21:40.420580 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:21:40.422711 (+ 2131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:40.423403 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:40.425786 (+ 2383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:40.426262 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:40.426365 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:21:40.430553 (+ 4188us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:21:40.430558 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:21:40.430562 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:21:40.430595 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:21:40.433448 (+ 2853us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:21:40.434282 (+ 834us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:21:40.434288 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:21:40.434292 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:21:40.434416 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:21:40.434849 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:21:40.434854 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:21:40.434857 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:21:40.435136 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:21:40.435292 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0620 20:21:40.435972 (+ 680us) server_negotiation.cc:299] Negotiation successful
0620 20:21:40.436197 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260620 20:21:40.437485 7726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:40.417267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:43175 (local address 127.0.0.1:48246)
0620 20:21:40.418134 (+ 867us) negotiation.cc:107] Waiting for socket to connect
0620 20:21:40.418180 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 20:21:40.420077 (+ 1897us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:21:40.420810 (+ 733us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:21:40.420824 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:21:40.421691 (+ 867us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:21:40.422518 (+ 827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:21:40.422545 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:21:40.423576 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:21:40.423583 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:21:40.425626 (+ 2043us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:21:40.425643 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:21:40.426504 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:21:40.426513 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:21:40.426827 (+ 314us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:21:40.427563 (+ 736us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:21:40.427601 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:21:40.430293 (+ 2692us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:21:40.433597 (+ 3304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:21:40.433606 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:21:40.433623 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:21:40.434153 (+ 530us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:21:40.434536 (+ 383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:21:40.434542 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:21:40.434545 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:21:40.434728 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:21:40.435266 (+ 538us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:21:40.435274 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:21:40.435612 (+ 338us) client_negotiation.cc:769] Sending connection context
0620 20:21:40.435876 (+ 264us) client_negotiation.cc:240] Negotiation successful
0620 20:21:40.436403 (+ 527us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":558,"thread_start_us":365,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.217.60@KRBTEST.COM" created.
I20260620 20:21:40.439291 7724 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.217.61" port: 33651 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 20:21:40 dist-test-slave-2hpg krb5kdc[7570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986900, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.217.61@KRBTEST.COM for kudu/127.16.217.61@KRBTEST.COM
I20260620 20:21:40.455014 7726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:40.440115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.61:33651 (local address 127.0.0.1:40296)
0620 20:21:40.440320 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0620 20:21:40.440341 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 20:21:40.440451 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:21:40.441182 (+ 731us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:21:40.441189 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:21:40.441722 (+ 533us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:21:40.441970 (+ 248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:21:40.441985 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:21:40.443315 (+ 1330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:21:40.443323 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:21:40.444808 (+ 1485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:21:40.444825 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:21:40.445657 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:21:40.445668 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:21:40.445905 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:21:40.446486 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:21:40.446521 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:21:40.448910 (+ 2389us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:21:40.452554 (+ 3644us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:21:40.452561 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:21:40.452566 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:21:40.452922 (+ 356us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:21:40.453298 (+ 376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:21:40.453304 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:21:40.453307 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:21:40.453450 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:21:40.454054 (+ 604us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:21:40.454060 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:21:40.454201 (+ 141us) client_negotiation.cc:769] Sending connection context
0620 20:21:40.454346 (+ 145us) client_negotiation.cc:240] Negotiation successful
0620 20:21:40.454601 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":63}
I20260620 20:21:40.455055 7728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:40.440207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40296 (local address 127.16.217.61:33651)
0620 20:21:40.440769 (+ 562us) server_negotiation.cc:206] Beginning negotiation
0620 20:21:40.440775 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:21:40.440799 (+ 24us) server_negotiation.cc:407] Connection header received
0620 20:21:40.440839 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:21:40.440845 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:21:40.441020 (+ 175us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:21:40.441127 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:21:40.442138 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:40.443173 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:40.444953 (+ 1780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:40.445475 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:40.445586 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:21:40.449118 (+ 3532us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:21:40.449132 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:21:40.449151 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:21:40.449195 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:21:40.452424 (+ 3229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:21:40.453072 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:21:40.453083 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:21:40.453085 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:21:40.453194 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:21:40.453565 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:21:40.453570 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:21:40.453573 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:21:40.453939 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:21:40.454150 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0620 20:21:40.454489 (+ 339us) server_negotiation.cc:299] Negotiation successful
0620 20:21:40.454694 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":418,"thread_start_us":274,"threads_started":1}
I20260620 20:21:40.457955 7724 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.217.60" port: 34865 } has no permanent_uuid. Determining permanent_uuid...
I20260620 20:21:40.459180 7726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:40.458700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.60:34865 (local address 127.0.0.1:44874)
0620 20:21:40.458890 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0620 20:21:40.459033 (+ 143us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.217.60:34865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
W20260620 20:21:40.459467 7675 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.60:34865: connect: Connection refused (error 111)
I20260620 20:21:40.461068 7726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:40.460705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.60:34865 (local address 127.0.0.1:44886)
0620 20:21:40.460871 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0620 20:21:40.460947 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.217.60:34865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260620 20:21:40.461305 7724 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.217.60:34865: Network error: Client connection negotiation failed: client connection to 127.16.217.60:34865: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.217.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.60.keytab.
Entry for principal kudu/127.16.217.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.60.keytab.
I20260620 20:21:40.473847 17252 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
/tmp/dist-test-taskQiX1sS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986536687152-17252-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986536687152-17252-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986536687152-17252-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986536687152-17252-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.217.60:34865
--webserver_interface=127.16.217.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.60.keytab
--principal=kudu/127.16.217.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.217.20:44807
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.217.62:43175,127.16.217.61:33651,127.16.217.60:34865
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986536687152-17252-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986536687152-17252-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986536687152-17252-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 20:21:40.510604 7724 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.217.60" port: 34865 } attempt: 1
I20260620 20:21:40.511829 7726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:40.511321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.60:34865 (local address 127.0.0.1:44898)
0620 20:21:40.511541 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0620 20:21:40.511667 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.217.60:34865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260620 20:21:40.512928 7726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:40.512557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.60:34865 (local address 127.0.0.1:44914)
0620 20:21:40.512742 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0620 20:21:40.512816 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.217.60:34865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260620 20:21:40.513139 7724 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.217.60:34865: Network error: Client connection negotiation failed: client connection to 127.16.217.60:34865: connect: Connection refused (error 111)
I20260620 20:21:40.581363 7724 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.217.60" port: 34865 } attempt: 2
I20260620 20:21:40.582640 7726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:40.582071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.60:34865 (local address 127.0.0.1:44924)
0620 20:21:40.582306 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0620 20:21:40.582436 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.217.60:34865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260620 20:21:40.583674 7726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:40.583317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.60:34865 (local address 127.0.0.1:44938)
0620 20:21:40.583449 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0620 20:21:40.583535 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.217.60:34865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260620 20:21:40.583894 7724 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.217.60:34865: Network error: Client connection negotiation failed: client connection to 127.16.217.60:34865: connect: Connection refused (error 111)
I20260620 20:21:40.588657 7650 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.217.61" port: 33651 } attempt: 5
Jun 20 20:21:40 dist-test-slave-2hpg krb5kdc[7570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986899, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.217.62@KRBTEST.COM for kudu/127.16.217.61@KRBTEST.COM
I20260620 20:21:40.603384 7652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:40.589315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.61:33651 (local address 127.0.0.1:40310)
0620 20:21:40.589531 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0620 20:21:40.589547 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 20:21:40.589647 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:21:40.590049 (+ 402us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:21:40.590052 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:21:40.590442 (+ 390us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:21:40.590670 (+ 228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:21:40.590680 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:21:40.592036 (+ 1356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:21:40.592039 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:21:40.592873 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:21:40.592883 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:21:40.593931 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:21:40.593944 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:21:40.594138 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:21:40.594774 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:21:40.594793 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:21:40.597145 (+ 2352us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:21:40.601010 (+ 3865us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:21:40.601014 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:21:40.601018 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:21:40.601419 (+ 401us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:21:40.601814 (+ 395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:21:40.601817 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:21:40.601820 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:21:40.602064 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:21:40.602678 (+ 614us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:21:40.602682 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:21:40.602819 (+ 137us) client_negotiation.cc:769] Sending connection context
0620 20:21:40.602930 (+ 111us) client_negotiation.cc:240] Negotiation successful
0620 20:21:40.603141 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56}
I20260620 20:21:40.603597 7728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:40.589382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40310 (local address 127.16.217.61:33651)
0620 20:21:40.589570 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0620 20:21:40.589576 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:21:40.589599 (+ 23us) server_negotiation.cc:407] Connection header received
0620 20:21:40.589768 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:21:40.589775 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:21:40.589893 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:21:40.590007 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:21:40.590819 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:40.591901 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:40.593024 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:40.593709 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:40.593832 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:21:40.597399 (+ 3567us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:21:40.597408 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:21:40.597416 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:21:40.597463 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:21:40.600862 (+ 3399us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:21:40.601542 (+ 680us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:21:40.601548 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:21:40.601551 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:21:40.601701 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:21:40.602171 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:21:40.602176 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:21:40.602179 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:21:40.602549 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:21:40.602707 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0620 20:21:40.603074 (+ 367us) server_negotiation.cc:299] Negotiation successful
0620 20:21:40.603298 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260620 20:21:40.604708 7650 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.217.60" port: 34865 } has no permanent_uuid. Determining permanent_uuid...
I20260620 20:21:40.605986 7652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:40.605467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.60:34865 (local address 127.0.0.1:44944)
0620 20:21:40.605671 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0620 20:21:40.605782 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.217.60:34865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260620 20:21:40.607102 7652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:40.606708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.60:34865 (local address 127.0.0.1:44958)
0620 20:21:40.606891 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0620 20:21:40.606986 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.217.60:34865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260620 20:21:40.607326 7650 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.217.60:34865: Network error: Client connection negotiation failed: client connection to 127.16.217.60:34865: connect: Connection refused (error 111)
I20260620 20:21:40.658586 7650 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.217.60" port: 34865 } attempt: 1
I20260620 20:21:40.660058 7652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:40.659355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.60:34865 (local address 127.0.0.1:44968)
0620 20:21:40.659647 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0620 20:21:40.659799 (+ 152us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.217.60:34865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
I20260620 20:21:40.661140 7652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:40.660777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.60:34865 (local address 127.0.0.1:44978)
0620 20:21:40.660925 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0620 20:21:40.661020 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.217.60:34865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260620 20:21:40.661438 7650 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.217.60:34865: Network error: Client connection negotiation failed: client connection to 127.16.217.60:34865: connect: Connection refused (error 111)
I20260620 20:21:40.675150 7724 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.217.60" port: 34865 } attempt: 3
I20260620 20:21:40.676326 7726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:40.675780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.60:34865 (local address 127.0.0.1:44992)
0620 20:21:40.676001 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0620 20:21:40.676122 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.217.60:34865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260620 20:21:40.677465 7726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:40.676992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.60:34865 (local address 127.0.0.1:45008)
0620 20:21:40.677239 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0620 20:21:40.677330 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.217.60:34865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128}
W20260620 20:21:40.677711 7724 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.217.60:34865: Network error: Client connection negotiation failed: client connection to 127.16.217.60:34865: connect: Connection refused (error 111)
I20260620 20:21:40.729704 7650 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.217.60" port: 34865 } attempt: 2
I20260620 20:21:40.730989 7652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:40.730396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.60:34865 (local address 127.0.0.1:45012)
0620 20:21:40.730641 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0620 20:21:40.730764 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.217.60:34865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260620 20:21:40.732187 7652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:40.731776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.60:34865 (local address 127.0.0.1:45022)
0620 20:21:40.731934 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0620 20:21:40.732037 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.217.60:34865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260620 20:21:40.732434 7650 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.217.60:34865: Network error: Client connection negotiation failed: client connection to 127.16.217.60:34865: connect: Connection refused (error 111)
W20260620 20:21:40.805009 7731 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 20:21:40.805367 7731 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:21:40.805486 7731 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:21:40.805620 7731 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:21:40.812870 7731 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:21:40.812966 7731 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:21:40.813060 7731 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:21:40.813134 7731 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:21:40.813246 7731 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:21:40.820943 7724 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.217.60" port: 34865 } attempt: 4
I20260620 20:21:40.822184 7726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:40.821586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.60:34865 (local address 127.0.0.1:45034)
0620 20:21:40.821849 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0620 20:21:40.821979 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.217.60:34865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260620 20:21:40.823191 7726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:40.822831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.60:34865 (local address 127.0.0.1:45048)
0620 20:21:40.822977 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0620 20:21:40.823057 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.217.60:34865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260620 20:21:40.823388 7724 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.217.60:34865: Network error: Client connection negotiation failed: client connection to 127.16.217.60:34865: connect: Connection refused (error 111)
I20260620 20:21:40.824096 7731 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.217.20:44807
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986536687152-17252-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986536687152-17252-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.16.217.62:43175,127.16.217.61:33651,127.16.217.60:34865
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986536687152-17252-0/krb5kdc/kudu_127.16.217.60.keytab
--principal=kudu/127.16.217.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.217.60:34865
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986536687152-17252-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.16.217.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986536687152-17252-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:21:40.825363 7731 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:21:40.826742 7731 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:21:40.838399 7739 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:21:40.838666 7650 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.217.60" port: 34865 } attempt: 3
W20260620 20:21:40.838428 7736 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:21:40.839304 7731 server_base.cc:1034] running on GCE node
W20260620 20:21:40.838353 7737 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:21:40.839807 7652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:40.839308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.60:34865 (local address 127.0.0.1:45062)
0620 20:21:40.839547 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0620 20:21:40.839644 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.217.60:34865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260620 20:21:40.840317 7731 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:21:40.840837 7652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:40.840475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.60:34865 (local address 127.0.0.1:45066)
0620 20:21:40.840664 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0620 20:21:40.840741 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.217.60:34865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
W20260620 20:21:40.841055 7650 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.217.60:34865: Network error: Client connection negotiation failed: client connection to 127.16.217.60:34865: connect: Connection refused (error 111)
I20260620 20:21:40.841802 7731 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:21:40.843080 7731 hybrid_clock.cc:648] HybridClock initialized: now 1781986900843022 us; error 31 us; skew 500 ppm
I20260620 20:21:40.843420 7731 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:21:40 dist-test-slave-2hpg krb5kdc[7570](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986900, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.217.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:21:40.846633 7731 init.cc:377] Logged in from keytab as kudu/127.16.217.60@KRBTEST.COM (short username kudu)
I20260620 20:21:40.848110 7731 webserver.cc:460] Webserver started at http://127.16.217.60:37529/ using document root <none> and password file <none>
I20260620 20:21:40.848771 7731 fs_manager.cc:359] Metadata directory not provided
I20260620 20:21:40.848918 7731 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:21:40.849233 7731 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:21:40.852380 7731 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986536687152-17252-0/minicluster-data/master-2/data/instance:
uuid: "75013c3abac943c195937e0d446b16af"
format_stamp: "Formatted at 2026-06-20 20:21:40 on dist-test-slave-2hpg"
server_key: "3998464c25ab22755dc59b02355e2cdd"
server_key_iv: "0cd306b9842447d13e0e15b4945cd630"
server_key_version: "encryptionkey@0"
I20260620 20:21:40.853142 7731 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986536687152-17252-0/minicluster-data/master-2/wal/instance:
uuid: "75013c3abac943c195937e0d446b16af"
format_stamp: "Formatted at 2026-06-20 20:21:40 on dist-test-slave-2hpg"
server_key: "3998464c25ab22755dc59b02355e2cdd"
server_key_iv: "0cd306b9842447d13e0e15b4945cd630"
server_key_version: "encryptionkey@0"
I20260620 20:21:40.858413 7731 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.006s
I20260620 20:21:40.861763 7746 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:21:40.863029 7731 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 20:21:40.863206 7731 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986536687152-17252-0/minicluster-data/master-2/data,/tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986536687152-17252-0/minicluster-data/master-2/wal
uuid: "75013c3abac943c195937e0d446b16af"
format_stamp: "Formatted at 2026-06-20 20:21:40 on dist-test-slave-2hpg"
server_key: "3998464c25ab22755dc59b02355e2cdd"
server_key_iv: "0cd306b9842447d13e0e15b4945cd630"
server_key_version: "encryptionkey@0"
I20260620 20:21:40.863410 7731 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986536687152-17252-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986536687152-17252-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986536687152-17252-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:21:40.887524 7731 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:21:40.892028 7731 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:21:40.892308 7731 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:21:40.910928 7731 rpc_server.cc:307] RPC server started. Bound to: 127.16.217.60:34865
I20260620 20:21:40.910946 7797 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.217.60:34865 every 8 connection(s)
I20260620 20:21:40.912225 7731 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986536687152-17252-0/minicluster-data/master-2/data/info.pb
I20260620 20:21:40.915980 7798 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:21:40.918898 17252 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu as pid 7731
I20260620 20:21:40.919095 17252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQiX1sS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986536687152-17252-0/minicluster-data/master-2/wal/instance
I20260620 20:21:40.919624 17252 external_mini_cluster.cc:1428] Setting key 13b26c660f81085f77efb1281f7406f7
I20260620 20:21:40.924005 7798 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.217.62" port: 43175 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 20:21:40 dist-test-slave-2hpg krb5kdc[7570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986899, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.217.60@KRBTEST.COM
Jun 20 20:21:40 dist-test-slave-2hpg krb5kdc[7570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986900, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.217.60@KRBTEST.COM for kudu/127.16.217.62@KRBTEST.COM
I20260620 20:21:40.941972 7653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:40.927008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48252 (local address 127.16.217.62:43175)
0620 20:21:40.927236 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0620 20:21:40.927243 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:21:40.927718 (+ 475us) server_negotiation.cc:407] Connection header received
0620 20:21:40.927919 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:21:40.927925 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:21:40.928002 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:21:40.928105 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:21:40.929191 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:40.929919 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:40.931280 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:40.931722 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:40.931856 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:21:40.935524 (+ 3668us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:21:40.935529 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:21:40.935533 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:21:40.935567 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:21:40.939108 (+ 3541us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:21:40.939831 (+ 723us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:21:40.939836 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:21:40.939838 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:21:40.939975 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:21:40.940380 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:21:40.940385 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:21:40.940389 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:21:40.940826 (+ 437us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:21:40.940993 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0620 20:21:40.941470 (+ 477us) server_negotiation.cc:299] Negotiation successful
0620 20:21:40.941639 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 20:21:40.942602 7801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:40.921516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45080 (local address 127.16.217.60:34865)
0620 20:21:40.922423 (+ 907us) server_negotiation.cc:206] Beginning negotiation
0620 20:21:40.922435 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 20:21:40.922476 (+ 41us) server_negotiation.cc:407] Connection header received
0620 20:21:40.924244 (+ 1768us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:21:40.924265 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:21:40.924694 (+ 429us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:21:40.925086 (+ 392us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:21:40.926826 (+ 1740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:40.928548 (+ 1722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:40.929774 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:40.930511 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:40.930700 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:21:40.935177 (+ 4477us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:21:40.935189 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:21:40.935203 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:21:40.935238 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:21:40.937944 (+ 2706us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:21:40.938861 (+ 917us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:21:40.938868 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:21:40.938874 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:21:40.939036 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:21:40.939466 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:21:40.939470 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:21:40.939473 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:21:40.940095 (+ 622us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:21:40.940392 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0620 20:21:40.940774 (+ 382us) server_negotiation.cc:299] Negotiation successful
0620 20:21:40.941181 (+ 407us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":705,"thread_start_us":461,"threads_started":1}
I20260620 20:21:40.942652 7802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:40.926965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.62:43175 (local address 127.0.0.1:48252)
0620 20:21:40.927582 (+ 617us) negotiation.cc:107] Waiting for socket to connect
0620 20:21:40.927615 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0620 20:21:40.927790 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:21:40.928163 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:21:40.928169 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:21:40.928791 (+ 622us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:21:40.929046 (+ 255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:21:40.929055 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:21:40.930080 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:21:40.930085 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:21:40.931144 (+ 1059us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:21:40.931156 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:21:40.931942 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:21:40.931951 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:21:40.932120 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:21:40.932705 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:21:40.932725 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:21:40.935268 (+ 2543us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:21:40.939222 (+ 3954us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:21:40.939229 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:21:40.939234 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:21:40.939718 (+ 484us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:21:40.940088 (+ 370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:21:40.940092 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:21:40.940096 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:21:40.940259 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:21:40.940971 (+ 712us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:21:40.940978 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:21:40.941142 (+ 164us) client_negotiation.cc:769] Sending connection context
0620 20:21:40.941351 (+ 209us) client_negotiation.cc:240] Negotiation successful
0620 20:21:40.941654 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":414,"thread_start_us":280,"threads_started":1}
I20260620 20:21:40.944535 7798 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.217.61" port: 33651 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 20:21:40 dist-test-slave-2hpg krb5kdc[7570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986900, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.217.60@KRBTEST.COM for kudu/127.16.217.61@KRBTEST.COM
Jun 20 20:21:40 dist-test-slave-2hpg krb5kdc[7570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986899, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.217.62@KRBTEST.COM
I20260620 20:21:40.959565 7802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:40.945660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.61:33651 (local address 127.0.0.1:40324)
0620 20:21:40.945939 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0620 20:21:40.945972 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0620 20:21:40.946106 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:21:40.946613 (+ 507us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:21:40.946618 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:21:40.947224 (+ 606us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:21:40.947413 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:21:40.947425 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:21:40.948422 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:21:40.948426 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:21:40.949394 (+ 968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:21:40.949407 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:21:40.950292 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:21:40.950303 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:21:40.950493 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:21:40.951130 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:21:40.951145 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:21:40.953398 (+ 2253us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:21:40.957019 (+ 3621us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:21:40.957023 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:21:40.957026 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:21:40.957498 (+ 472us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:21:40.957996 (+ 498us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:21:40.957999 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:21:40.958001 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:21:40.958136 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:21:40.958752 (+ 616us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:21:40.958756 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:21:40.958865 (+ 109us) client_negotiation.cc:769] Sending connection context
0620 20:21:40.959000 (+ 135us) client_negotiation.cc:240] Negotiation successful
0620 20:21:40.959180 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151}
I20260620 20:21:40.959604 7728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:40.945672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40324 (local address 127.16.217.61:33651)
0620 20:21:40.946127 (+ 455us) server_negotiation.cc:206] Beginning negotiation
0620 20:21:40.946169 (+ 42us) server_negotiation.cc:399] Waiting for connection header
0620 20:21:40.946196 (+ 27us) server_negotiation.cc:407] Connection header received
0620 20:21:40.946288 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:21:40.946296 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:21:40.946410 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:21:40.946549 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:21:40.947571 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:40.948257 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:40.949578 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:40.950124 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:40.950236 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:21:40.953579 (+ 3343us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:21:40.953584 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:21:40.953588 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:21:40.953618 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:21:40.956885 (+ 3267us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:21:40.957641 (+ 756us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:21:40.957648 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:21:40.957652 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:21:40.957818 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:21:40.958251 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:21:40.958257 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:21:40.958260 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:21:40.958581 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:21:40.958750 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0620 20:21:40.959120 (+ 370us) server_negotiation.cc:299] Negotiation successful
0620 20:21:40.959350 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147}
I20260620 20:21:40.960611 7798 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.217.60" port: 34865 } has no permanent_uuid. Determining permanent_uuid...
I20260620 20:21:40.962047 7653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:40.946367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48264 (local address 127.16.217.62:43175)
0620 20:21:40.946573 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0620 20:21:40.946578 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:21:40.946595 (+ 17us) server_negotiation.cc:407] Connection header received
0620 20:21:40.946923 (+ 328us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:21:40.946927 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:21:40.946984 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:21:40.947165 (+ 181us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:21:40.948393 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:40.949120 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:40.950732 (+ 1612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:40.951288 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:40.951381 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:21:40.955656 (+ 4275us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:21:40.955664 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:21:40.955671 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:21:40.955712 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:21:40.959152 (+ 3440us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:21:40.959997 (+ 845us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:21:40.960001 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:21:40.960003 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:21:40.960222 (+ 219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:21:40.960590 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:21:40.960595 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:21:40.960599 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:21:40.960917 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:21:40.961210 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0620 20:21:40.961471 (+ 261us) server_negotiation.cc:299] Negotiation successful
0620 20:21:40.961699 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
Jun 20 20:21:40 dist-test-slave-2hpg krb5kdc[7570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986900, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.217.60@KRBTEST.COM for kudu/127.16.217.60@KRBTEST.COM
Jun 20 20:21:40 dist-test-slave-2hpg krb5kdc[7570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986899, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.217.61@KRBTEST.COM
I20260620 20:21:40.973563 7801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:40.961601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45090 (local address 127.16.217.60:34865)
0620 20:21:40.961759 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0620 20:21:40.961764 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:21:40.961777 (+ 13us) server_negotiation.cc:407] Connection header received
0620 20:21:40.961816 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:21:40.961854 (+ 38us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:21:40.961947 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:21:40.962031 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:21:40.962656 (+ 625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:40.963510 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:40.964604 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:40.965262 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:40.965408 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:21:40.968327 (+ 2919us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:21:40.968332 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:21:40.968335 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:21:40.968361 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:21:40.970884 (+ 2523us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:21:40.971560 (+ 676us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:21:40.971565 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:21:40.971570 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:21:40.971705 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:21:40.972065 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:21:40.972072 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:21:40.972075 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:21:40.972433 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:21:40.972561 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0620 20:21:40.973001 (+ 440us) server_negotiation.cc:299] Negotiation successful
0620 20:21:40.973273 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":58}
I20260620 20:21:40.973953 7802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:40.961386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.60:34865 (local address 127.0.0.1:45090)
0620 20:21:40.961537 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0620 20:21:40.961555 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 20:21:40.961661 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:21:40.962047 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:21:40.962051 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:21:40.962401 (+ 350us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:21:40.962528 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:21:40.962535 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:21:40.963647 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:21:40.963654 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:21:40.964471 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:21:40.964481 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:21:40.965402 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:21:40.965405 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:21:40.965561 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:21:40.966101 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:21:40.966116 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:21:40.968157 (+ 2041us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:21:40.971020 (+ 2863us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:21:40.971023 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:21:40.971026 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:21:40.971352 (+ 326us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:21:40.971822 (+ 470us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:21:40.971826 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:21:40.971828 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:21:40.971954 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:21:40.972562 (+ 608us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:21:40.972569 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:21:40.972748 (+ 179us) client_negotiation.cc:769] Sending connection context
0620 20:21:40.972926 (+ 178us) client_negotiation.cc:240] Negotiation successful
0620 20:21:40.973476 (+ 550us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52,"mutex_wait_us":213}
I20260620 20:21:40.978519 7728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:40.963606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40340 (local address 127.16.217.61:33651)
0620 20:21:40.963762 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0620 20:21:40.963767 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:21:40.963783 (+ 16us) server_negotiation.cc:407] Connection header received
0620 20:21:40.963921 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:21:40.963926 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:21:40.963993 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:21:40.964088 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:21:40.965472 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:40.966353 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:40.967588 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:40.968094 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:40.968221 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:21:40.972449 (+ 4228us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:21:40.972454 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:21:40.972457 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:21:40.972485 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:21:40.975906 (+ 3421us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:21:40.976512 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:21:40.976519 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:21:40.976523 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:21:40.976675 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:21:40.977114 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:21:40.977119 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:21:40.977122 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:21:40.977455 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:21:40.977730 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0620 20:21:40.977982 (+ 252us) server_negotiation.cc:299] Negotiation successful
0620 20:21:40.978220 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 20:21:40.979907 7798 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 75013c3abac943c195937e0d446b16af: Bootstrap starting.
I20260620 20:21:40.983505 17252 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 20:21:40.983752 7798 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 75013c3abac943c195937e0d446b16af: Neither blocks nor log segments found. Creating new log.
I20260620 20:21:40.984997 7798 log.cc:824] T 00000000000000000000000000000000 P 75013c3abac943c195937e0d446b16af: Log is configured to *not* fsync() on all Append() calls
I20260620 20:21:40.987812 7798 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 75013c3abac943c195937e0d446b16af: No bootstrap required, opened a new log
I20260620 20:21:40.986639 7625 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "fb0dd4cdadea49458ae5e99c8f666997"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48264
I20260620 20:21:40.994967 7798 raft_consensus.cc:348] T 00000000000000000000000000000000 P 75013c3abac943c195937e0d446b16af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb0dd4cdadea49458ae5e99c8f666997" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 43175 } } peers { permanent_uuid: "38c990bb69be4a4488cce8c5bbccb219" member_type: VOTER last_known_addr { host: "127.16.217.61" port: 33651 } } peers { permanent_uuid: "75013c3abac943c195937e0d446b16af" member_type: VOTER last_known_addr { host: "127.16.217.60" port: 34865 } }
I20260620 20:21:40.995296 7798 raft_consensus.cc:729] T 00000000000000000000000000000000 P 75013c3abac943c195937e0d446b16af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75013c3abac943c195937e0d446b16af, State: Initialized, Role: FOLLOWER
I20260620 20:21:40.995743 7798 consensus_queue.cc:260] T 00000000000000000000000000000000 P 75013c3abac943c195937e0d446b16af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb0dd4cdadea49458ae5e99c8f666997" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 43175 } } peers { permanent_uuid: "38c990bb69be4a4488cce8c5bbccb219" member_type: VOTER last_known_addr { host: "127.16.217.61" port: 33651 } } peers { permanent_uuid: "75013c3abac943c195937e0d446b16af" member_type: VOTER last_known_addr { host: "127.16.217.60" port: 34865 } }
I20260620 20:21:40.996636 7804 sys_catalog.cc:455] T 00000000000000000000000000000000 P 75013c3abac943c195937e0d446b16af [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb0dd4cdadea49458ae5e99c8f666997" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 43175 } } peers { permanent_uuid: "38c990bb69be4a4488cce8c5bbccb219" member_type: VOTER last_known_addr { host: "127.16.217.61" port: 33651 } } peers { permanent_uuid: "75013c3abac943c195937e0d446b16af" member_type: VOTER last_known_addr { host: "127.16.217.60" port: 34865 } } }
I20260620 20:21:40.997025 7804 sys_catalog.cc:458] T 00000000000000000000000000000000 P 75013c3abac943c195937e0d446b16af [sys.catalog]: This master's current role is: FOLLOWER
I20260620 20:21:40.997594 7798 sys_catalog.cc:564] T 00000000000000000000000000000000 P 75013c3abac943c195937e0d446b16af [sys.catalog]: configured and running, proceeding with master startup.
W20260620 20:21:41.007215 7815 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 75013c3abac943c195937e0d446b16af: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 20:21:41.007375 7815 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 20:21:41.005038 7731 master_runner.cc:186] Error getting master registration for 127.16.217.62:43175: OK, instance_id { permanent_uuid: "fb0dd4cdadea49458ae5e99c8f666997" instance_seqno: 1781986899878961 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 20:21:41.016534 7731 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 20:21:41.018237 7650 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.217.60" port: 34865 } attempt: 4
Jun 20 20:21:41 dist-test-slave-2hpg krb5kdc[7570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986899, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.217.62@KRBTEST.COM for kudu/127.16.217.60@KRBTEST.COM
I20260620 20:21:41.034478 7801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:41.019009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45094 (local address 127.16.217.60:34865)
0620 20:21:41.019296 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0620 20:21:41.019302 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:21:41.019326 (+ 24us) server_negotiation.cc:407] Connection header received
0620 20:21:41.019449 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:21:41.019454 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:21:41.019532 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:21:41.019685 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:21:41.020676 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:41.021835 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:41.023761 (+ 1926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:41.024492 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:41.024622 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:21:41.028706 (+ 4084us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:21:41.028711 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:21:41.028714 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:21:41.028748 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:21:41.031700 (+ 2952us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:21:41.032447 (+ 747us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:21:41.032455 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:21:41.032458 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:21:41.032630 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:21:41.033109 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:21:41.033117 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:21:41.033120 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:21:41.033462 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:21:41.033590 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0620 20:21:41.034032 (+ 442us) server_negotiation.cc:299] Negotiation successful
0620 20:21:41.034226 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
I20260620 20:21:41.034668 7652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:41.018940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.60:34865 (local address 127.0.0.1:45094)
0620 20:21:41.019172 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0620 20:21:41.019194 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 20:21:41.019322 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:21:41.019737 (+ 415us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:21:41.019742 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:21:41.020265 (+ 523us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:21:41.020518 (+ 253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:21:41.020533 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:21:41.021995 (+ 1462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:21:41.022001 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:21:41.023589 (+ 1588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:21:41.023607 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:21:41.024664 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:21:41.024672 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:21:41.024888 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:21:41.025596 (+ 708us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:21:41.025622 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:21:41.028515 (+ 2893us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:21:41.031847 (+ 3332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:21:41.031852 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:21:41.031857 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:21:41.032284 (+ 427us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:21:41.032761 (+ 477us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:21:41.032767 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:21:41.032772 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:21:41.032952 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:21:41.033568 (+ 616us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:21:41.033573 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:21:41.033765 (+ 192us) client_negotiation.cc:769] Sending connection context
0620 20:21:41.033958 (+ 193us) client_negotiation.cc:240] Negotiation successful
0620 20:21:41.034274 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":77}
I20260620 20:21:41.039305 7650 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb0dd4cdadea49458ae5e99c8f666997: Bootstrap starting.
I20260620 20:21:41.042603 7650 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fb0dd4cdadea49458ae5e99c8f666997: Neither blocks nor log segments found. Creating new log.
I20260620 20:21:41.043677 7650 log.cc:824] T 00000000000000000000000000000000 P fb0dd4cdadea49458ae5e99c8f666997: Log is configured to *not* fsync() on all Append() calls
I20260620 20:21:41.046609 7650 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb0dd4cdadea49458ae5e99c8f666997: No bootstrap required, opened a new log
I20260620 20:21:41.052605 7650 raft_consensus.cc:348] T 00000000000000000000000000000000 P fb0dd4cdadea49458ae5e99c8f666997 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb0dd4cdadea49458ae5e99c8f666997" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 43175 } } peers { permanent_uuid: "38c990bb69be4a4488cce8c5bbccb219" member_type: VOTER last_known_addr { host: "127.16.217.61" port: 33651 } } peers { permanent_uuid: "75013c3abac943c195937e0d446b16af" member_type: VOTER last_known_addr { host: "127.16.217.60" port: 34865 } }
I20260620 20:21:41.052963 7650 raft_consensus.cc:729] T 00000000000000000000000000000000 P fb0dd4cdadea49458ae5e99c8f666997 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb0dd4cdadea49458ae5e99c8f666997, State: Initialized, Role: FOLLOWER
I20260620 20:21:41.053396 7650 consensus_queue.cc:260] T 00000000000000000000000000000000 P fb0dd4cdadea49458ae5e99c8f666997 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb0dd4cdadea49458ae5e99c8f666997" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 43175 } } peers { permanent_uuid: "38c990bb69be4a4488cce8c5bbccb219" member_type: VOTER last_known_addr { host: "127.16.217.61" port: 33651 } } peers { permanent_uuid: "75013c3abac943c195937e0d446b16af" member_type: VOTER last_known_addr { host: "127.16.217.60" port: 34865 } }
I20260620 20:21:41.054371 7818 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb0dd4cdadea49458ae5e99c8f666997 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb0dd4cdadea49458ae5e99c8f666997" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 43175 } } peers { permanent_uuid: "38c990bb69be4a4488cce8c5bbccb219" member_type: VOTER last_known_addr { host: "127.16.217.61" port: 33651 } } peers { permanent_uuid: "75013c3abac943c195937e0d446b16af" member_type: VOTER last_known_addr { host: "127.16.217.60" port: 34865 } } }
I20260620 20:21:41.054742 7818 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb0dd4cdadea49458ae5e99c8f666997 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 20:21:41.055521 7650 sys_catalog.cc:564] T 00000000000000000000000000000000 P fb0dd4cdadea49458ae5e99c8f666997 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:21:41.055675 7625 raft_consensus.cc:482] T 00000000000000000000000000000000 P fb0dd4cdadea49458ae5e99c8f666997 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260620 20:21:41.055886 7625 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fb0dd4cdadea49458ae5e99c8f666997 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:21:41.059449 7625 raft_consensus.cc:504] T 00000000000000000000000000000000 P fb0dd4cdadea49458ae5e99c8f666997 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb0dd4cdadea49458ae5e99c8f666997" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 43175 } } peers { permanent_uuid: "38c990bb69be4a4488cce8c5bbccb219" member_type: VOTER last_known_addr { host: "127.16.217.61" port: 33651 } } peers { permanent_uuid: "75013c3abac943c195937e0d446b16af" member_type: VOTER last_known_addr { host: "127.16.217.60" port: 34865 } }
I20260620 20:21:41.062083 7625 leader_election.cc:288] T 00000000000000000000000000000000 P fb0dd4cdadea49458ae5e99c8f666997 [CANDIDATE]: Term 1 election: Requested vote from peers 38c990bb69be4a4488cce8c5bbccb219 (127.16.217.61:33651), 75013c3abac943c195937e0d446b16af (127.16.217.60:34865)
I20260620 20:21:41.062672 7773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "fb0dd4cdadea49458ae5e99c8f666997" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "75013c3abac943c195937e0d446b16af"
I20260620 20:21:41.063165 7773 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 75013c3abac943c195937e0d446b16af [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:21:41.067469 7773 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 75013c3abac943c195937e0d446b16af [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fb0dd4cdadea49458ae5e99c8f666997 in term 1.
I20260620 20:21:41.068220 7601 leader_election.cc:302] T 00000000000000000000000000000000 P fb0dd4cdadea49458ae5e99c8f666997 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 75013c3abac943c195937e0d446b16af, fb0dd4cdadea49458ae5e99c8f666997; no voters:
I20260620 20:21:41.068851 7818 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fb0dd4cdadea49458ae5e99c8f666997 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:21:41.075475 7818 raft_consensus.cc:686] T 00000000000000000000000000000000 P fb0dd4cdadea49458ae5e99c8f666997 [term 1 LEADER]: Becoming Leader. State: Replica: fb0dd4cdadea49458ae5e99c8f666997, State: Running, Role: LEADER
I20260620 20:21:41.076248 7818 consensus_queue.cc:237] T 00000000000000000000000000000000 P fb0dd4cdadea49458ae5e99c8f666997 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb0dd4cdadea49458ae5e99c8f666997" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 43175 } } peers { permanent_uuid: "38c990bb69be4a4488cce8c5bbccb219" member_type: VOTER last_known_addr { host: "127.16.217.61" port: 33651 } } peers { permanent_uuid: "75013c3abac943c195937e0d446b16af" member_type: VOTER last_known_addr { host: "127.16.217.60" port: 34865 } }
I20260620 20:21:41.081722 7832 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb0dd4cdadea49458ae5e99c8f666997 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fb0dd4cdadea49458ae5e99c8f666997. Latest consensus state: current_term: 1 leader_uuid: "fb0dd4cdadea49458ae5e99c8f666997" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb0dd4cdadea49458ae5e99c8f666997" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 43175 } } peers { permanent_uuid: "38c990bb69be4a4488cce8c5bbccb219" member_type: VOTER last_known_addr { host: "127.16.217.61" port: 33651 } } peers { permanent_uuid: "75013c3abac943c195937e0d446b16af" member_type: VOTER last_known_addr { host: "127.16.217.60" port: 34865 } } }
I20260620 20:21:41.082358 7832 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb0dd4cdadea49458ae5e99c8f666997 [sys.catalog]: This master's current role is: LEADER
I20260620 20:21:41.085335 7583 master_runner.cc:186] Error getting master registration for 127.16.217.61:33651: OK, instance_id { permanent_uuid: "38c990bb69be4a4488cce8c5bbccb219" instance_seqno: 1781986900394118 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 20:21:41.100934 7583 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260620 20:21:41.104787 7834 catalog_manager.cc:1558] T 00000000000000000000000000000000 P fb0dd4cdadea49458ae5e99c8f666997: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 20:21:41.105000 7834 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 20:21:41.105953 7835 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 20:21:41.108062 7835 catalog_manager.cc:1484] Initializing Kudu cluster ID...
W20260620 20:21:41.118501 7604 tablet.cc:2369] T 00000000000000000000000000000000 P fb0dd4cdadea49458ae5e99c8f666997: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 20:21:41.122588 7724 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.217.60" port: 34865 } attempt: 5
I20260620 20:21:41.132457 7773 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 75013c3abac943c195937e0d446b16af [term 1 FOLLOWER]: Refusing update from remote peer fb0dd4cdadea49458ae5e99c8f666997: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:21:41.134032 7832 consensus_queue.cc:1035] T 00000000000000000000000000000000 P fb0dd4cdadea49458ae5e99c8f666997 [LEADER]: Connected to new peer: Peer: permanent_uuid: "75013c3abac943c195937e0d446b16af" member_type: VOTER last_known_addr { host: "127.16.217.60" port: 34865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Jun 20 20:21:41 dist-test-slave-2hpg krb5kdc[7570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986900, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.217.61@KRBTEST.COM for kudu/127.16.217.60@KRBTEST.COM
W20260620 20:21:41.144459 7752 tablet.cc:2369] T 00000000000000000000000000000000 P 75013c3abac943c195937e0d446b16af: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 20:21:41.146586 7801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:41.107381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45108 (local address 127.16.217.60:34865)
0620 20:21:41.107617 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0620 20:21:41.107626 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 20:21:41.124331 (+ 16705us) server_negotiation.cc:407] Connection header received
0620 20:21:41.124548 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:21:41.124554 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:21:41.124612 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:21:41.124681 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:21:41.129060 (+ 4379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:41.130126 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:41.132997 (+ 2871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:41.133723 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:41.133831 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:21:41.137285 (+ 3454us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:21:41.137295 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:21:41.137299 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:21:41.137337 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:21:41.141617 (+ 4280us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:21:41.143324 (+ 1707us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:21:41.143331 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:21:41.143335 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:21:41.143511 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:21:41.143999 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:21:41.144006 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:21:41.144011 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:21:41.144418 (+ 407us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:21:41.144719 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0620 20:21:41.145403 (+ 684us) server_negotiation.cc:299] Negotiation successful
0620 20:21:41.145673 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260620 20:21:41.152741 7726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:41.123194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.60:34865 (local address 127.0.0.1:45122)
0620 20:21:41.123365 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0620 20:21:41.123384 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 20:21:41.123481 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:21:41.124795 (+ 1314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:21:41.124802 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:21:41.125355 (+ 553us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:21:41.125585 (+ 230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:21:41.125599 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:21:41.128493 (+ 2894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:21:41.128499 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:21:41.129907 (+ 1408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:21:41.129922 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:21:41.131560 (+ 1638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:21:41.131566 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:21:41.131816 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:21:41.132551 (+ 735us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:21:41.132581 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:21:41.136075 (+ 3494us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:21:41.148901 (+ 12826us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:21:41.148906 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:21:41.148910 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:21:41.149498 (+ 588us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:21:41.150741 (+ 1243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:21:41.150749 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:21:41.150753 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:21:41.150935 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:21:41.151630 (+ 695us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:21:41.151635 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:21:41.151816 (+ 181us) client_negotiation.cc:769] Sending connection context
0620 20:21:41.151977 (+ 161us) client_negotiation.cc:240] Negotiation successful
0620 20:21:41.152333 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20260620 20:21:41.153981 7842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:41.123255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45122 (local address 127.16.217.60:34865)
0620 20:21:41.124224 (+ 969us) server_negotiation.cc:206] Beginning negotiation
0620 20:21:41.124255 (+ 31us) server_negotiation.cc:399] Waiting for connection header
0620 20:21:41.124285 (+ 30us) server_negotiation.cc:407] Connection header received
0620 20:21:41.124407 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:21:41.124413 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:21:41.124549 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:21:41.125814 (+ 1265us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:21:41.126372 (+ 558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:41.128277 (+ 1905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:41.130128 (+ 1851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:41.131307 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:41.132243 (+ 936us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:21:41.136584 (+ 4341us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:21:41.136596 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:21:41.136602 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:21:41.136684 (+ 82us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:21:41.148760 (+ 12076us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:21:41.150382 (+ 1622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:21:41.150389 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:21:41.150394 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:21:41.150592 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:21:41.151095 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:21:41.151099 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:21:41.151102 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:21:41.151484 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:21:41.153121 (+ 1637us) server_negotiation.cc:1035] Waiting for connection context
0620 20:21:41.153271 (+ 150us) server_negotiation.cc:299] Negotiation successful
0620 20:21:41.153496 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":754,"thread_start_us":510,"threads_started":1}
I20260620 20:21:41.153821 7804 sys_catalog.cc:455] T 00000000000000000000000000000000 P 75013c3abac943c195937e0d446b16af [sys.catalog]: SysCatalogTable state changed. Reason: New leader fb0dd4cdadea49458ae5e99c8f666997. Latest consensus state: current_term: 1 leader_uuid: "fb0dd4cdadea49458ae5e99c8f666997" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb0dd4cdadea49458ae5e99c8f666997" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 43175 } } peers { permanent_uuid: "38c990bb69be4a4488cce8c5bbccb219" member_type: VOTER last_known_addr { host: "127.16.217.61" port: 33651 } } peers { permanent_uuid: "75013c3abac943c195937e0d446b16af" member_type: VOTER last_known_addr { host: "127.16.217.60" port: 34865 } } }
I20260620 20:21:41.154882 7804 sys_catalog.cc:458] T 00000000000000000000000000000000 P 75013c3abac943c195937e0d446b16af [sys.catalog]: This master's current role is: FOLLOWER
I20260620 20:21:41.159760 7724 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 38c990bb69be4a4488cce8c5bbccb219: Bootstrap starting.
I20260620 20:21:41.164801 7724 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 38c990bb69be4a4488cce8c5bbccb219: Neither blocks nor log segments found. Creating new log.
I20260620 20:21:41.166384 7724 log.cc:824] T 00000000000000000000000000000000 P 38c990bb69be4a4488cce8c5bbccb219: Log is configured to *not* fsync() on all Append() calls
I20260620 20:21:41.167600 7832 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb0dd4cdadea49458ae5e99c8f666997 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "fb0dd4cdadea49458ae5e99c8f666997" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb0dd4cdadea49458ae5e99c8f666997" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 43175 } } peers { permanent_uuid: "38c990bb69be4a4488cce8c5bbccb219" member_type: VOTER last_known_addr { host: "127.16.217.61" port: 33651 } } peers { permanent_uuid: "75013c3abac943c195937e0d446b16af" member_type: VOTER last_known_addr { host: "127.16.217.60" port: 34865 } } }
I20260620 20:21:41.167930 7832 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb0dd4cdadea49458ae5e99c8f666997 [sys.catalog]: This master's current role is: LEADER
I20260620 20:21:41.171512 7835 catalog_manager.cc:1347] Generated new cluster ID: b7468040baca4a36b498498e4d56c2ea
I20260620 20:21:41.171648 7835 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 20:21:41.172971 7724 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 38c990bb69be4a4488cce8c5bbccb219: No bootstrap required, opened a new log
I20260620 20:21:41.182261 7804 sys_catalog.cc:455] T 00000000000000000000000000000000 P 75013c3abac943c195937e0d446b16af [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "fb0dd4cdadea49458ae5e99c8f666997" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb0dd4cdadea49458ae5e99c8f666997" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 43175 } } peers { permanent_uuid: "38c990bb69be4a4488cce8c5bbccb219" member_type: VOTER last_known_addr { host: "127.16.217.61" port: 33651 } } peers { permanent_uuid: "75013c3abac943c195937e0d446b16af" member_type: VOTER last_known_addr { host: "127.16.217.60" port: 34865 } } }
I20260620 20:21:41.182548 7804 sys_catalog.cc:458] T 00000000000000000000000000000000 P 75013c3abac943c195937e0d446b16af [sys.catalog]: This master's current role is: FOLLOWER
I20260620 20:21:41.184518 7724 raft_consensus.cc:348] T 00000000000000000000000000000000 P 38c990bb69be4a4488cce8c5bbccb219 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb0dd4cdadea49458ae5e99c8f666997" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 43175 } } peers { permanent_uuid: "38c990bb69be4a4488cce8c5bbccb219" member_type: VOTER last_known_addr { host: "127.16.217.61" port: 33651 } } peers { permanent_uuid: "75013c3abac943c195937e0d446b16af" member_type: VOTER last_known_addr { host: "127.16.217.60" port: 34865 } }
I20260620 20:21:41.184950 7724 raft_consensus.cc:729] T 00000000000000000000000000000000 P 38c990bb69be4a4488cce8c5bbccb219 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 38c990bb69be4a4488cce8c5bbccb219, State: Initialized, Role: FOLLOWER
I20260620 20:21:41.185550 7724 consensus_queue.cc:260] T 00000000000000000000000000000000 P 38c990bb69be4a4488cce8c5bbccb219 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb0dd4cdadea49458ae5e99c8f666997" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 43175 } } peers { permanent_uuid: "38c990bb69be4a4488cce8c5bbccb219" member_type: VOTER last_known_addr { host: "127.16.217.61" port: 33651 } } peers { permanent_uuid: "75013c3abac943c195937e0d446b16af" member_type: VOTER last_known_addr { host: "127.16.217.60" port: 34865 } }
I20260620 20:21:41.192797 7653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:41.103269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48274 (local address 127.16.217.62:43175)
0620 20:21:41.103450 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0620 20:21:41.103456 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:21:41.118565 (+ 15109us) server_negotiation.cc:407] Connection header received
0620 20:21:41.118884 (+ 319us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:21:41.118891 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:21:41.118982 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:21:41.119076 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:21:41.125346 (+ 6270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:41.126481 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:41.150284 (+ 23803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:41.151064 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:41.151178 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:21:41.167325 (+ 16147us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:21:41.167335 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:21:41.167340 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:21:41.167389 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:21:41.171199 (+ 3810us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:21:41.189527 (+ 18328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:21:41.189535 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:21:41.189539 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:21:41.189739 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:21:41.190268 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:21:41.190274 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:21:41.190278 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:21:41.190652 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:21:41.190906 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0620 20:21:41.192129 (+ 1223us) server_negotiation.cc:299] Negotiation successful
0620 20:21:41.192411 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 20:21:41.192611 7848 sys_catalog.cc:455] T 00000000000000000000000000000000 P 38c990bb69be4a4488cce8c5bbccb219 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb0dd4cdadea49458ae5e99c8f666997" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 43175 } } peers { permanent_uuid: "38c990bb69be4a4488cce8c5bbccb219" member_type: VOTER last_known_addr { host: "127.16.217.61" port: 33651 } } peers { permanent_uuid: "75013c3abac943c195937e0d446b16af" member_type: VOTER last_known_addr { host: "127.16.217.60" port: 34865 } } }
I20260620 20:21:41.193032 7848 sys_catalog.cc:458] T 00000000000000000000000000000000 P 38c990bb69be4a4488cce8c5bbccb219 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 20:21:41.195384 7728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:41.103377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40342 (local address 127.16.217.61:33651)
0620 20:21:41.103570 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0620 20:21:41.103578 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 20:21:41.121934 (+ 18356us) server_negotiation.cc:407] Connection header received
0620 20:21:41.122199 (+ 265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:21:41.122206 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:21:41.122343 (+ 137us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:21:41.122419 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:21:41.145830 (+ 23411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:41.147031 (+ 1201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:41.155286 (+ 8255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:41.156017 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:41.156130 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:21:41.184648 (+ 28518us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:21:41.184657 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:21:41.184662 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:21:41.184707 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:21:41.188609 (+ 3902us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:21:41.191389 (+ 2780us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:21:41.191395 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:21:41.191399 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:21:41.191561 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:21:41.193549 (+ 1988us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:21:41.193555 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:21:41.193558 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:21:41.193940 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:21:41.194195 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0620 20:21:41.194764 (+ 569us) server_negotiation.cc:299] Negotiation successful
0620 20:21:41.195003 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 20:21:41.195020 7724 sys_catalog.cc:564] T 00000000000000000000000000000000 P 38c990bb69be4a4488cce8c5bbccb219 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:21:41.201670 7698 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 38c990bb69be4a4488cce8c5bbccb219 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:21:41.206105 7699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "fb0dd4cdadea49458ae5e99c8f666997" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "38c990bb69be4a4488cce8c5bbccb219"
I20260620 20:21:41.208253 7698 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 38c990bb69be4a4488cce8c5bbccb219 [term 1 FOLLOWER]: Refusing update from remote peer fb0dd4cdadea49458ae5e99c8f666997: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:21:41.213022 7699 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 38c990bb69be4a4488cce8c5bbccb219 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fb0dd4cdadea49458ae5e99c8f666997 in term 1.
I20260620 20:21:41.216135 7818 consensus_queue.cc:1035] T 00000000000000000000000000000000 P fb0dd4cdadea49458ae5e99c8f666997 [LEADER]: Connected to new peer: Peer: permanent_uuid: "38c990bb69be4a4488cce8c5bbccb219" member_type: VOTER last_known_addr { host: "127.16.217.61" port: 33651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:21:41.240513 7835 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 20:21:41.242517 7835 catalog_manager.cc:1504] Loading token signing keys...
I20260620 20:21:41.257395 7848 sys_catalog.cc:455] T 00000000000000000000000000000000 P 38c990bb69be4a4488cce8c5bbccb219 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fb0dd4cdadea49458ae5e99c8f666997. Latest consensus state: current_term: 1 leader_uuid: "fb0dd4cdadea49458ae5e99c8f666997" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb0dd4cdadea49458ae5e99c8f666997" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 43175 } } peers { permanent_uuid: "38c990bb69be4a4488cce8c5bbccb219" member_type: VOTER last_known_addr { host: "127.16.217.61" port: 33651 } } peers { permanent_uuid: "75013c3abac943c195937e0d446b16af" member_type: VOTER last_known_addr { host: "127.16.217.60" port: 34865 } } }
I20260620 20:21:41.257732 7848 sys_catalog.cc:458] T 00000000000000000000000000000000 P 38c990bb69be4a4488cce8c5bbccb219 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 20:21:41.258054 7832 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb0dd4cdadea49458ae5e99c8f666997 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "fb0dd4cdadea49458ae5e99c8f666997" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb0dd4cdadea49458ae5e99c8f666997" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 43175 } } peers { permanent_uuid: "38c990bb69be4a4488cce8c5bbccb219" member_type: VOTER last_known_addr { host: "127.16.217.61" port: 33651 } } peers { permanent_uuid: "75013c3abac943c195937e0d446b16af" member_type: VOTER last_known_addr { host: "127.16.217.60" port: 34865 } } }
I20260620 20:21:41.258380 7832 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb0dd4cdadea49458ae5e99c8f666997 [sys.catalog]: This master's current role is: LEADER
I20260620 20:21:41.258277 7857 sys_catalog.cc:455] T 00000000000000000000000000000000 P 38c990bb69be4a4488cce8c5bbccb219 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "fb0dd4cdadea49458ae5e99c8f666997" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb0dd4cdadea49458ae5e99c8f666997" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 43175 } } peers { permanent_uuid: "38c990bb69be4a4488cce8c5bbccb219" member_type: VOTER last_known_addr { host: "127.16.217.61" port: 33651 } } peers { permanent_uuid: "75013c3abac943c195937e0d446b16af" member_type: VOTER last_known_addr { host: "127.16.217.60" port: 34865 } } }
I20260620 20:21:41.258793 7857 sys_catalog.cc:458] T 00000000000000000000000000000000 P 38c990bb69be4a4488cce8c5bbccb219 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 20:21:41.272859 7835 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fb0dd4cdadea49458ae5e99c8f666997: Generated new TSK 0
I20260620 20:21:41.274415 7835 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 20:21:41.330826 7865 catalog_manager.cc:1259] Loaded cluster ID: b7468040baca4a36b498498e4d56c2ea
I20260620 20:21:41.331033 7865 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 38c990bb69be4a4488cce8c5bbccb219: loading cluster ID for follower catalog manager: success
I20260620 20:21:41.334940 7865 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 38c990bb69be4a4488cce8c5bbccb219: acquiring CA information for follower catalog manager: success
I20260620 20:21:41.337570 7865 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 38c990bb69be4a4488cce8c5bbccb219: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260620 20:21:41.432586 7653 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:21:41.407550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48278 (local address 127.16.217.62:43175)
0620 20:21:41.407782 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0620 20:21:41.407789 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:21:41.430376 (+ 22587us) server_negotiation.cc:407] Connection header received
0620 20:21:41.430908 (+ 532us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:21:41.430917 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:21:41.431025 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:21:41.431121 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:21:41.432311 (+ 1190us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48278: BlockingRecv error: recv got EOF from 127.0.0.1:48278 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
W20260620 20:21:41.436079 7842 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:21:41.425019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45138 (local address 127.16.217.60:34865)
0620 20:21:41.425214 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0620 20:21:41.425221 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:21:41.433910 (+ 8689us) server_negotiation.cc:407] Connection header received
0620 20:21:41.434377 (+ 467us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:21:41.434386 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:21:41.434501 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:21:41.434584 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:21:41.435771 (+ 1187us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45138: BlockingRecv error: recv got EOF from 127.0.0.1:45138 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
W20260620 20:21:41.438371 7728 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:21:41.414353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40358 (local address 127.16.217.61:33651)
0620 20:21:41.414568 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0620 20:21:41.414575 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:21:41.433490 (+ 18915us) server_negotiation.cc:407] Connection header received
0620 20:21:41.436622 (+ 3132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:21:41.436632 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:21:41.436774 (+ 142us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:21:41.436862 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:21:41.438042 (+ 1180us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40358: BlockingRecv error: recv got EOF from 127.0.0.1:40358 (error 108)
Metrics: {"server-negotiator.queue_time_us":86}
I20260620 20:21:41.491706 7728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:41.472176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40364 (local address 127.16.217.61:33651)
0620 20:21:41.472393 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0620 20:21:41.472402 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 20:21:41.483295 (+ 10893us) server_negotiation.cc:407] Connection header received
0620 20:21:41.483928 (+ 633us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:21:41.483938 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:21:41.484168 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:21:41.484322 (+ 154us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 20:21:41.485145 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:41.486353 (+ 1208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:41.488511 (+ 2158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:41.489164 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:41.489273 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:21:41.490123 (+ 850us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 20:21:41.490845 (+ 722us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 20:21:41.490978 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0620 20:21:41.491220 (+ 242us) server_negotiation.cc:299] Negotiation successful
0620 20:21:41.491329 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
W20260620 20:21:41.534796 7842 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:21:41.524781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45142 (local address 127.16.217.60:34865)
0620 20:21:41.524981 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0620 20:21:41.524988 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:21:41.532469 (+ 7481us) server_negotiation.cc:407] Connection header received
0620 20:21:41.533066 (+ 597us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:21:41.533074 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:21:41.533203 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:21:41.533296 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:21:41.534493 (+ 1197us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45142: BlockingRecv error: recv got EOF from 127.0.0.1:45142 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
I20260620 20:21:41.580081 7728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:41.563695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40366 (local address 127.16.217.61:33651)
0620 20:21:41.563895 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0620 20:21:41.563903 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 20:21:41.572705 (+ 8802us) server_negotiation.cc:407] Connection header received
0620 20:21:41.573239 (+ 534us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:21:41.573262 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:21:41.573389 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:21:41.573476 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 20:21:41.574225 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:41.575523 (+ 1298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:41.577597 (+ 2074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:41.578313 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:41.578420 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:21:41.579128 (+ 708us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 20:21:41.579320 (+ 192us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 20:21:41.579391 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 20:21:41.579665 (+ 274us) server_negotiation.cc:299] Negotiation successful
0620 20:21:41.579760 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
W20260620 20:21:41.623327 7842 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:21:41.613320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45154 (local address 127.16.217.60:34865)
0620 20:21:41.613539 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0620 20:21:41.613546 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:21:41.621331 (+ 7785us) server_negotiation.cc:407] Connection header received
0620 20:21:41.621829 (+ 498us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:21:41.621836 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:21:41.621974 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:21:41.622059 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:21:41.623088 (+ 1029us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45154: BlockingRecv error: recv got EOF from 127.0.0.1:45154 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
I20260620 20:21:41.677100 7728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:41.661677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40374 (local address 127.16.217.61:33651)
0620 20:21:41.661880 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0620 20:21:41.661887 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:21:41.670534 (+ 8647us) server_negotiation.cc:407] Connection header received
0620 20:21:41.670735 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:21:41.670742 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:21:41.670835 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:21:41.670935 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 20:21:41.671620 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:41.672712 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:41.674759 (+ 2047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:41.675460 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:41.675556 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:21:41.676198 (+ 642us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 20:21:41.676353 (+ 155us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 20:21:41.676418 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0620 20:21:41.676657 (+ 239us) server_negotiation.cc:299] Negotiation successful
0620 20:21:41.676753 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
W20260620 20:21:41.709496 7842 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:21:41.707042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45162 (local address 127.16.217.60:34865)
0620 20:21:41.707235 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0620 20:21:41.707244 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 20:21:41.708112 (+ 868us) server_negotiation.cc:407] Connection header received
0620 20:21:41.708338 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:21:41.708346 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:21:41.708440 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:21:41.708553 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:21:41.709300 (+ 747us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45162: BlockingRecv error: recv got EOF from 127.0.0.1:45162 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
I20260620 20:21:41.724995 7728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:41.719444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40376 (local address 127.16.217.61:33651)
0620 20:21:41.719695 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0620 20:21:41.719703 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 20:21:41.720072 (+ 369us) server_negotiation.cc:407] Connection header received
0620 20:21:41.720280 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:21:41.720286 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:21:41.720364 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:21:41.720465 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 20:21:41.720883 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:41.721701 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:41.723078 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:41.723744 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:41.723888 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:21:41.724294 (+ 406us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 20:21:41.724397 (+ 103us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 20:21:41.724500 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 20:21:41.724649 (+ 149us) server_negotiation.cc:299] Negotiation successful
0620 20:21:41.724729 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111}
W20260620 20:21:41.733242 7842 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:21:41.731152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45174 (local address 127.16.217.60:34865)
0620 20:21:41.731438 (+ 286us) server_negotiation.cc:206] Beginning negotiation
0620 20:21:41.731447 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 20:21:41.731902 (+ 455us) server_negotiation.cc:407] Connection header received
0620 20:21:41.732104 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:21:41.732109 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:21:41.732194 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:21:41.732307 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:21:41.733022 (+ 715us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45174: BlockingRecv error: recv got EOF from 127.0.0.1:45174 (error 108)
Metrics: {"server-negotiator.queue_time_us":125}
I20260620 20:21:41.750250 7728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:41.744486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40388 (local address 127.16.217.61:33651)
0620 20:21:41.744745 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0620 20:21:41.744754 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 20:21:41.745265 (+ 511us) server_negotiation.cc:407] Connection header received
0620 20:21:41.745482 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:21:41.745496 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:21:41.745575 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:21:41.745672 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 20:21:41.746149 (+ 477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:41.746894 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:41.748267 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:41.748770 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:41.748927 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:21:41.749396 (+ 469us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 20:21:41.749508 (+ 112us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 20:21:41.749609 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0620 20:21:41.749735 (+ 126us) server_negotiation.cc:299] Negotiation successful
0620 20:21:41.749814 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
W20260620 20:21:41.766521 7842 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:21:41.764013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45176 (local address 127.16.217.60:34865)
0620 20:21:41.764259 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0620 20:21:41.764268 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 20:21:41.764865 (+ 597us) server_negotiation.cc:407] Connection header received
0620 20:21:41.765087 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:21:41.765093 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:21:41.765235 (+ 142us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:21:41.765344 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:21:41.766337 (+ 993us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45176: BlockingRecv error: recv got EOF from 127.0.0.1:45176 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
I20260620 20:21:41.792065 7728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:41.786466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40392 (local address 127.16.217.61:33651)
0620 20:21:41.786688 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0620 20:21:41.786697 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 20:21:41.787248 (+ 551us) server_negotiation.cc:407] Connection header received
0620 20:21:41.787479 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:21:41.787484 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:21:41.787560 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:21:41.787712 (+ 152us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 20:21:41.788084 (+ 372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:41.788890 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:41.790290 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:41.790732 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:41.790869 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:21:41.791303 (+ 434us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 20:21:41.791414 (+ 111us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 20:21:41.791523 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0620 20:21:41.791662 (+ 139us) server_negotiation.cc:299] Negotiation successful
0620 20:21:41.791749 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
W20260620 20:21:41.800197 7842 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:21:41.798026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45190 (local address 127.16.217.60:34865)
0620 20:21:41.798273 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0620 20:21:41.798282 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 20:21:41.798779 (+ 497us) server_negotiation.cc:407] Connection header received
0620 20:21:41.798986 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:21:41.798992 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:21:41.799089 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:21:41.799194 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:21:41.799929 (+ 735us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45190: BlockingRecv error: recv got EOF from 127.0.0.1:45190 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
I20260620 20:21:41.841675 7728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:41.835558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40408 (local address 127.16.217.61:33651)
0620 20:21:41.835765 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0620 20:21:41.835773 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 20:21:41.836082 (+ 309us) server_negotiation.cc:407] Connection header received
0620 20:21:41.836250 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:21:41.836256 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:21:41.836385 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:21:41.836502 (+ 117us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 20:21:41.836960 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:41.838102 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:41.839381 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:41.840187 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:41.840319 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:21:41.840742 (+ 423us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 20:21:41.840879 (+ 137us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 20:21:41.841003 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0620 20:21:41.841183 (+ 180us) server_negotiation.cc:299] Negotiation successful
0620 20:21:41.841303 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
W20260620 20:21:41.850190 7842 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:21:41.848016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45204 (local address 127.16.217.60:34865)
0620 20:21:41.848235 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0620 20:21:41.848242 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:21:41.848560 (+ 318us) server_negotiation.cc:407] Connection header received
0620 20:21:41.848764 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:21:41.848770 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:21:41.848877 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:21:41.849000 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:21:41.849949 (+ 949us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45204: BlockingRecv error: recv got EOF from 127.0.0.1:45204 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 20:21:41.923827 7728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:41.917557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40420 (local address 127.16.217.61:33651)
0620 20:21:41.917746 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0620 20:21:41.917752 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:21:41.918319 (+ 567us) server_negotiation.cc:407] Connection header received
0620 20:21:41.918519 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:21:41.918524 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:21:41.918594 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:21:41.918668 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 20:21:41.919190 (+ 522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:41.920121 (+ 931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:41.922122 (+ 2001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:41.922556 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:41.922654 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:21:41.923088 (+ 434us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 20:21:41.923198 (+ 110us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 20:21:41.923262 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0620 20:21:41.923431 (+ 169us) server_negotiation.cc:299] Negotiation successful
0620 20:21:41.923502 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
W20260620 20:21:41.932839 7842 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:21:41.930601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45208 (local address 127.16.217.60:34865)
0620 20:21:41.930773 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0620 20:21:41.930780 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:21:41.931241 (+ 461us) server_negotiation.cc:407] Connection header received
0620 20:21:41.931396 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:21:41.931400 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:21:41.931495 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:21:41.931581 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:21:41.932548 (+ 967us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45208: BlockingRecv error: recv got EOF from 127.0.0.1:45208 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 20:21:42.009598 7815 catalog_manager.cc:1259] Loaded cluster ID: b7468040baca4a36b498498e4d56c2ea
I20260620 20:21:42.009707 7815 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 75013c3abac943c195937e0d446b16af: loading cluster ID for follower catalog manager: success
I20260620 20:21:42.012189 7815 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 75013c3abac943c195937e0d446b16af: acquiring CA information for follower catalog manager: success
I20260620 20:21:42.013952 7815 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 75013c3abac943c195937e0d446b16af: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 20:21:42.069422 7728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:42.064243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40430 (local address 127.16.217.61:33651)
0620 20:21:42.064435 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0620 20:21:42.064439 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:21:42.064766 (+ 327us) server_negotiation.cc:407] Connection header received
0620 20:21:42.064925 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:21:42.064930 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:21:42.064998 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:21:42.065093 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 20:21:42.065486 (+ 393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:42.066282 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:42.067864 (+ 1582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:42.068335 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:42.068430 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:21:42.068822 (+ 392us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 20:21:42.068926 (+ 104us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 20:21:42.068990 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0620 20:21:42.069146 (+ 156us) server_negotiation.cc:299] Negotiation successful
0620 20:21:42.069208 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 20:21:42.082235 7842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:42.076396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45216 (local address 127.16.217.60:34865)
0620 20:21:42.076638 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0620 20:21:42.076644 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:21:42.076971 (+ 327us) server_negotiation.cc:407] Connection header received
0620 20:21:42.077109 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:21:42.077113 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:21:42.077274 (+ 161us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:21:42.077375 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 20:21:42.077777 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:42.078579 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:42.079876 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:42.080323 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:42.080434 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:21:42.080917 (+ 483us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 20:21:42.081540 (+ 623us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 20:21:42.081669 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0620 20:21:42.081804 (+ 135us) server_negotiation.cc:299] Negotiation successful
0620 20:21:42.081937 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260620 20:21:42.098824 7728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:42.088940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40432 (local address 127.16.217.61:33651)
0620 20:21:42.089194 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0620 20:21:42.089202 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 20:21:42.091669 (+ 2467us) server_negotiation.cc:407] Connection header received
0620 20:21:42.091880 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:21:42.091887 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:21:42.091986 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:21:42.092074 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 20:21:42.093138 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:42.094137 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:42.096125 (+ 1988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:42.096856 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:42.096987 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:21:42.097584 (+ 597us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 20:21:42.097701 (+ 117us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 20:21:42.097818 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0620 20:21:42.098416 (+ 598us) server_negotiation.cc:299] Negotiation successful
0620 20:21:42.098554 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 20:21:42.099354 7973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:42.088603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48288 (local address 127.16.217.62:43175)
0620 20:21:42.090718 (+ 2115us) server_negotiation.cc:206] Beginning negotiation
0620 20:21:42.090723 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:21:42.091424 (+ 701us) server_negotiation.cc:407] Connection header received
0620 20:21:42.091539 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:21:42.091542 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:21:42.091803 (+ 261us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:21:42.092499 (+ 696us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 20:21:42.093029 (+ 530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:42.094047 (+ 1018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:42.096330 (+ 2283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:42.096967 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:42.097093 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:21:42.098243 (+ 1150us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 20:21:42.098700 (+ 457us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 20:21:42.098825 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0620 20:21:42.098937 (+ 112us) server_negotiation.cc:299] Negotiation successful
0620 20:21:42.099020 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1941,"thread_start_us":1795,"threads_started":1}
I20260620 20:21:42.099680 7842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:42.088479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45226 (local address 127.16.217.60:34865)
0620 20:21:42.088737 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0620 20:21:42.088746 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 20:21:42.092461 (+ 3715us) server_negotiation.cc:407] Connection header received
0620 20:21:42.092673 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:21:42.092677 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:21:42.092766 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:21:42.092870 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 20:21:42.093543 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:42.094688 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:42.096805 (+ 2117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:42.097443 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:42.097580 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:21:42.098094 (+ 514us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 20:21:42.098248 (+ 154us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 20:21:42.098340 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0620 20:21:42.099308 (+ 968us) server_negotiation.cc:299] Negotiation successful
0620 20:21:42.099396 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260620 20:21:42.109784 17252 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu pid 7583
I20260620 20:21:42.110987 7645 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:21:42.222091 17252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu with pid 7583
W20260620 20:21:42.243800 7748 connection.cc:535] server connection from 127.0.0.1:45094 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 20:21:42.244192 17252 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu pid 7657
I20260620 20:21:42.245380 7719 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:21:42.352511 17252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu with pid 7657
I20260620 20:21:42.372879 17252 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu pid 7731
I20260620 20:21:42.374138 7793 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:21:42.481096 17252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQiX1sS/build/asan/bin/kudu with pid 7731
2026-06-20T20:21:42Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (4413 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (8745 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (365755 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST